Image depth information generation method, device and storage medium

Abstract

The present invention discloses a method, device and storage medium for generating image depth information, and relates to the field of image processing technology, in order to solve the problem that when training a monocular depth estimation model, the reprojection loss calculation of the occluded area of ​​the image is inaccurate, and the depth estimation of the background area at the edge of the foreground object is too small. The image depth information generation method comprises: obtaining an image to be tested; processing the image to be tested according to a monocular depth estimation model to determine the depth information; the monocular depth estimation model is a model trained based on a binocular image pair, and the objective function of the monocular depth estimation model is used to at least characterize the corrected reprojection error of the binocular image pair, and the corrected reprojection error is the reprojection error of filtering out inaccurate pixels. The image depth information generation method provided by the present invention is used to obtain clearer object boundaries when estimating the depth map corresponding to a single image, thereby improving the overall performance of the monocular depth estimation model.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device and storage medium for generating image depth information. Background Art

[0002] When using a binocular image pair to train a depth estimation model, due to the influence of object occlusion in stereo space, that is, some points in the environment are only visible in one image of the binocular image pair, the calculation of the reprojection loss of the occluded area of ​​the image is inaccurate.

[0003] For the above reasons, the model obtained by the above training tends to underestimate the depth of the background area at the edge of the foreground object, which is manifested in the depth map output by the model as blurred boundaries of the foreground object and larger apparent size of the object. Summary of the invention

[0004] The purpose of the present invention is to provide a method for generating image depth information, which is used to solve the technical problem in the prior art that when training a monocular depth estimation model, the reprojection loss of the occluded area of ​​the image is inaccurately calculated, and the depth of the background area at the edge of the foreground object is estimated to be too small.

[0005] In order to achieve the above object, the present invention provides a method for generating image depth information, comprising:

[0006] Acquire an image to be tested; process the image to be tested according to a monocular depth estimation model to determine depth information; wherein the monocular depth estimation model is a model trained based on a binocular image pair, and an objective function of the monocular depth estimation model is determined by at least a corrected reprojection error characterizing the binocular image pair, and the corrected reprojection error is a reprojection error that filters out inaccurate pixels.

[0007] Compared with the prior art, in the image depth information generation method provided by the present invention, a binocular image pair is used to train a monocular depth estimation model, thereby avoiding the problem of dynamic objects caused by introducing monocular video training; in the model training, the reprojection process is combined to correct the reprojection error to quickly remove the influence of the reprojection error of the image occluded area on the calculation result, thereby solving the problem that when training a monocular depth estimation model in the prior art, the reprojection loss calculation of the image occluded area is inaccurate, resulting in a smaller depth estimation of the background area at the edge of the foreground object. Based on this, when the image depth information generation method provided by the present invention uses a monocular depth estimation model to process the image to be tested, the monocular depth estimation model can better estimate the depth of the object boundary, thereby improving the accuracy of the monocular depth estimation model, and can ensure that the object boundary presented by the determined depth information is clearer than the prior art, thereby improving the technical effect of the overall performance of the monocular depth estimation model.

[0008] The present invention also provides an image processing device, comprising:

[0009] A graphics processor and a communication interface coupled to the graphics processor; the graphics processor is used to run a computer program or instruction to implement the image depth information generation method described in the above technical solution.

[0010] Compared with the prior art, the beneficial effects of an image processing device provided by the present invention are the same as the beneficial effects of an image depth information generating method described in the above technical solution, which will not be elaborated here.

[0011] The present invention also provides a computer storage medium, in which instructions are stored. When the instructions are executed, the image depth information generating method described in the above technical solution is implemented.

[0012] Compared with the prior art, the beneficial effects of a computer storage medium provided by the present invention are the same as the beneficial effects of an image depth information generating method described in the above technical solution, and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0014] Figure 1 One of the structural block diagrams of the image processing device in an embodiment of the present invention;

[0015] Figure 2 This is a schematic diagram of a binocular camera generating a depth map according to a foreground object in an embodiment of the present invention;

[0016] Figure 3 One of the flowcharts of the method for generating image depth information in an embodiment of the present invention;

[0017] Figure 4 This is a second flowchart of the method for generating image depth information in an embodiment of the present invention;

[0018] Figure 5 A schematic diagram of a method for generating image depth information in accordance with an embodiment of the present invention for detecting an occluded area;

[0019] Figure 6 This is a flowchart of the method for generating image depth information according to an embodiment of the present invention;

[0020] Figure 7 A comparison between a depth map generated by the image depth information generation method in an embodiment of the present invention and depth maps generated by other algorithms;

[0021] Figure 8 This is the second structural block diagram of the image processing device in the embodiment of the present invention;

[0022] Fig. 9 is a hardware structure block diagram of an image processing device in an embodiment of the present invention;

[0023] Fig.10 Schematic diagram of the structure of the chip in the embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first threshold and the second threshold are only used to distinguish different thresholds, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0025] It should be noted that, in the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0026] In the present invention, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b, c can be single or multiple.

[0027] Before introducing the embodiments of the present invention, the following definitions are given for the relevant terms involved in the embodiments of the present invention:

[0028] Structural Similarity (SSIM) is a measure of the similarity between two images. SSIM uses two images, one is an uncompressed image without distortion, and the other is a distorted image. μ x is the average value of x, μ y is the mean value of y, is the variance of x, is the variance of y, σ xy is the covariance of x and y. x and y are the vectors of the two images respectively, and the value range of structural similarity is -1 to 1. When the two images are exactly the same, the value of SSIM is equal to 1.

[0029] The Adam algorithm (its name comes from adaptive moment estimation) is a first-order optimization algorithm that can replace the traditional stochastic gradient descent process. It can iteratively update the weights of the neural network based on the training data.

[0030] Encoder-Decoder is a very common model framework in deep learning. For example, the auto-encoding unsupervised algorithm is designed and trained using the encoder-decoder structure; for example, the image caption application, which has been popular in recent years, is the CNN-RNN (CNN, Convolutional Neural Networks; RNN, Recurrent Neural Network) encoding-decoding framework; another example is the neural network machine translation NMT model, which is often the LSTM-LSTM (LSTM, Long Short-Term Memory) encoding-decoding framework. Therefore, to be precise, Encoder-Decoder is not a specific model, but a type of framework. The Encoder and Decoder parts can be any text, voice, image, video data, and the model can use CNN, RNN, BiRNN (Bidirectional RNN, bidirectional recurrent neural network), LSTM, GRU (gated recurrent unit, gated recurrent unit), etc. Therefore, based on Encoder-Decoder, various application algorithms can be designed.

[0031] In the field of artificial intelligence and image processing, depth information, as a basic information for representing scene structure, is the basis for many applications such as 3D reconstruction, obstacle avoidance, and navigation. Depending on whether the depth information can be directly obtained using sensors, the methods for obtaining depth information can be divided into active and passive methods.

[0032] Active methods generally use RGB-D cameras or LiDAR. RGB-D cameras are generally only suitable for indoor scenes due to their limited measurement range. Although LiDAR has a larger measurement range and can basically meet the needs of outdoor scenes, the depth information it measures is particularly sparse compared to the camera resolution, and the high production cost of LiDAR limits its application.

[0033] Passive methods generally use the images obtained by the camera to perform a series of processing to obtain a depth map. According to the number of cameras, passive methods can be further divided into monocular, binocular and multi-objective depth estimation methods. Among them, binocular and multi-objective methods can be effectively solved using traditional multi-view stereo geometry methods. In order to use these methods to obtain high-precision dense depth maps, a large computational cost is required. The methods using a single camera for depth estimation can be divided into methods using monocular image sequences and methods using a single image. The methods using monocular image sequences often rely on accurate estimation of inter-frame motion, which is affected by moving objects and occlusions, and this estimation may not be accurate. The method based on the single image inference to obtain the corresponding depth map, that is, the monocular depth estimation method, does not need to rely on other cameras or images to directly infer the depth map.

[0034] Due to the ill-posedness of monocular depth estimation, it has been difficult to solve this problem well using manually designed features or clues. In recent years, with the rise of deep learning, researchers have begun to try to use convolutional neural networks to solve this problem. Early deep learning-based monocular depth estimation methods mainly used supervised training to obtain models, but because it is difficult to obtain the real depth map required for supervised training on a large scale, self-supervised methods based on reprojection errors have gradually become the mainstream of monocular depth estimation research. Self-supervised monocular depth estimation mainly uses monocular image sequences or binocular image pairs for training. When using monocular image sequences for training, the calculated reprojection loss will be too large because the destruction of the static scene assumption by moving objects has not been effectively solved, which in turn affects the performance of the model. Using binocular image pairs for training can avoid the influence of moving objects, thereby obtaining more accurate depth estimation.

[0035] As with monocular image sequences, when using binocular image pairs to train a monocular depth estimation model, there will be occlusion problems that make the reprojection error estimate too large. In the Monodepth2 model proposed by Clement Godard, monocular video is introduced to solve the occlusion problem when training with binocular graphics. However, the introduction of monocular video also brings the influence of moving objects, and the Auto-Masking technology proposed by the author cannot fully solve the influence of moving objects. Therefore, the present invention attempts to solve the occlusion problem in the monocular depth estimation model by using binocular image pair training without introducing monocular video.

[0036] In order to overcome the above technical problems, an embodiment of the present invention provides an image depth information generation method, which can obtain clearer object boundaries when estimating the depth map corresponding to a single image, thereby improving the overall performance of the monocular depth estimation model. The image depth information generation method can be applied to an image depth information generation system.

[0037] Figure 1 FIG. 1 is a schematic diagram of a depth map generation system provided by an embodiment of the present invention. Figure 1 As shown, the image depth information generation system includes: an image acquisition device and an image processing device.

[0038] The above-mentioned image acquisition device is used to acquire binocular image pairs. The image acquisition device 200 can be used to acquire monocular images, binocular image pairs, and the like. The image acquisition device 200 can be a calibrated binocular camera or a monocular camera.

[0039] The image acquisition device 200 can communicate with the image processing device 100, and is used to send the acquired binocular image pair or monocular image to the image processing device 100. The image processing device 100 performs processing according to the acquired picture.

[0040] like Figure 2 As shown in the figure, when using a binocular camera to collect a binocular image pair, C0 and C1 are the focal points of the left and right cameras, lr and lg represent the red foreground object and the green background object, respectively. Point G is a point on lg. The imaging point of point G in the left camera is point G0. Due to the occlusion of lr, point G is not visible in the right camera. If the model estimates the depth of pixel G0 correctly (L), the pixel value of the image synthesized by the right image and the depth map at point G0 is the pixel value of R1. At this time, the reprojection error at point G0 is the difference between the pixel values ​​of the two points |I 0,G0 -I 1,R1 |, the error is very large. Only when the network estimates the depth of G0 to be less than Lth (also less than the actual depth L), for example, the position of G0 is estimated to be G', then the reprojection error at G0 is |I 0,G0 -I 1,G1| is close to 0, which makes the reprojection loss of G0 pixels small.

[0041] An embodiment of the present invention provides a method for generating a depth map, which can be executed by an image processing device or applied to a chip of an image processing device. The method for generating a depth map provided by an embodiment of the present invention is described below with reference to the accompanying drawings.

[0042] Figure 3 The following is a schematic diagram illustrating a method for generating a depth map provided in an embodiment of the present application. Figure 3 As shown, the method for generating a depth map provided by an embodiment of the present invention includes:

[0043] Step S101: The image processing device acquires the image to be tested. In practical applications, the image collector may be a monocular camera, which sends the acquired image to the image processing device for image processing.

[0044] Step 102: The image processing device processes the image to be measured according to the monocular depth estimation model to determine a depth map.

[0045] In practical applications, the occlusion detection and reprojection process can be closely combined to solve the problem of inaccurate depth estimation around foreground objects caused by occlusion, thereby improving the performance of monocular depth estimation. Based on this, the monocular depth estimation model is a model based on binocular image pair training. That is to say, the training object of the monocular depth estimation model is a binocular image pair. At the same time, the objective function of the monocular depth estimation model is used to at least characterize the corrected reprojection error of the binocular image pair. The corrected reprojection error is the reprojection error that filters out inaccurate pixels. Since the calculation of the reprojection loss in the occluded area of ​​the image is inaccurate when performing traditional binocular image pair training, the corrected reprojection error in the embodiment of the present invention is the reprojection error that filters out inaccurate pixels. Based on this, when the image processing device processes the image to be tested according to the monocular depth estimation model, the points with poor depth estimation are filtered out, and the accuracy of the monocular depth estimation model is improved. Based on this, the technical effect of the overall performance of the monocular depth estimation model adopted by the method provided in the embodiment of the present invention is relatively good, and it can be ensured that the object boundary presented by the determined depth map is clearer than the prior art. The training process of the monocular depth estimation model can be found below.

[0046] The model adopted in the embodiment of the present invention uses binocular image pairs as training objects. During training, a binocular camera can be used to capture scene images to obtain a large number of binocular image pairs. Some transformations (rotation, perturbation, etc.) are performed on the binocular image pairs to expand the original training set. A loss function is used to learn the network model, and finally the ability of self-supervised learning is evaluated to achieve self-supervision of the training process.

[0047] Among them, machine learning is divided into two basic learning paradigms: supervised learning and unsupervised learning. Supervised learning uses a large amount of labeled data to train the model. After the loss between the model's prediction and the real label of the data, back propagation is performed, and finally the network model acquires the ability to recognize new samples. Unsupervised learning does not rely on any label value. It finds the relationship between samples by mining the intrinsic characteristics of the data, such as clustering-related tasks. The main difference between supervised and unsupervised is whether the network model requires manually annotated label information during training. Self-supervised learning mainly uses auxiliary tasks (pretext) to mine its own supervisory information from large-scale unsupervised data. The network model is trained through this constructed supervisory information, so that it can learn representations that are valuable for downstream tasks.

[0048] When training a monocular depth estimation model, such as Figure 4 and Figure 5 As shown in the figure, the binocular image pair data is first collected synchronously using the binocular camera. When collecting, the left and right cameras are set at the same horizontal height and with the same orientation angle. The camera internal parameter matrix K is known, and the relative positions of the left and right cameras are T lr and T rl , the resolution of the binocular image pairs captured by the left and right cameras is W0×H0. After the left and right cameras synchronously capture enough RGB image pairs, they are saved for use in training the monocular depth estimation model. The monocular image is captured by a monocular camera, and the acquisition operation is also performed after the camera position is set and the internal parameter matrix of the camera is obtained. It should be understood that the images used to train the monocular depth estimation model can be image pairs captured by the image acquisition device 200, or they can be image data sets of training data sets and test data sets. The training data set and the test data set contain multiple training pictures and test pictures, and each picture corresponds to a binocular image pair.

[0049] The left camera in the above binocular camera can be designated as the target camera, and the right camera as the source camera. Of course, the left camera can also be designated as the source camera, and the right camera as the target camera. Based on this, the binocular image pair includes a target image and a source image. The following description takes the left camera as the target camera and the right camera as the source camera.

[0050] The setting parameters of the above binocular camera include the camera internal parameter matrix K and the relative position T of the left and right cameras. lr and T rl , T lr is the relative position of the left camera relative to the right camera, T rl is the relative position of the left camera relative to the right camera. The camera focal points of the left and right cameras are C t and C s, the distant object and the near object are imaged on the left image to obtain the target image, and the distant object and the near object are imaged on the right image to obtain the source image. The target image and the source image are a binocular image pair.

[0051] When the left camera is the source camera and the right camera is the target camera, the image captured by the left camera is the source image and the image captured by the right camera is the target image. When training binocular image pairs, different monocular depth estimation models are selected based on system performance and real-time performance, such as input image resolution, optimization objective function and other parameters.

[0052] Figure 4 The following is a flowchart showing how to train a monocular depth estimation model using a pair of binocular images. Figure 4 As shown in the figure, the encoder-decoder structure is used as the network structure for binocular image pair training. The network structure is based on U-Net (a type of fully convolutional neural network). In order to achieve a lightweight network structure, ResNet18 (ResNet, residual network, residual neural network, proposed by He Kaiming, Zhang Xiangyu and others from Microsoft Research, ResNet18 refers to 18 layers with weights, including 17 convolutional layers and 1 fully connected layer) is selected as the encoder. The resolution of the input image of the network is 640×192. A deeper network (such as ResNet50 as the encoder) and a larger resolution can also be used.

[0053] When the size W0×H0 of the binocular image pair is different from the input image resolution W×H of the monocular depth estimation model, the size of the binocular image pair needs to be scaled during binocular image pair training so that the size of the binocular image pair is consistent with the input image resolution W×H of the monocular depth estimation model. In addition, the two images of the binocular image pair can be flipped left and right and / or random perturbations can be added with a certain probability to simulate the effects of object flipping, illumination changes, etc. on the image, thereby improving the training performance of the monocular depth estimation model; and the original training set can be expanded to improve the generalization ability of the model.

[0054] Based on the above image preprocessing, the preprocessed images are used for image training, and the objective function of the monocular depth estimation model can be determined by at least the reprojection error of the binocular image pair. The reprojection error of the binocular image pair is determined by the L1 error and structural similarity between the synthesized image and the target image. The synthesized image is determined by the depth map of the target image and the source image. Specifically, the reprojection error of the binocular image pair can be obtained in the following manner:

[0055] The first step is to select a suitable monocular depth estimation model, input the target image in the binocular image pair into the network model, and output n depth maps D with different resolutions. i(i=0,1,…,n-1), i is the serial number of the depth map. The resolution of the i-th depth map is Using the depth map D 0i (i=0,1,…,n-1) and the source image I in the binocular image pair s (The corresponding camera is called the source camera) synthesizes the target image This process is called reprojection.

[0056] Among them, according to the mapping relationship between the three-dimensional coordinates and the source camera coordinate system, the pixel points in the target image are converted between homogeneous coordinates and non-homogeneous coordinates, and the pixel points in the target image are converted to the coordinates in the source camera coordinate system. Then, bilinear interpolation is used to calculate the pixel value of the point where the coordinate is located in the source image, and the pixel value is used as an approximate estimate of the pixel in the target image.

[0057] For example, for each pixel coordinate (u, v) of the target image, 0≤u≤W-1,0≤v≤H-1, the pixel corresponds to the depth map D 0i The depth d uv =D oi (u,v) is the three-dimensional coordinate point P in the target camera coordinate system t =(x,y,z) T Projected, the three-dimensional coordinate point P in the target camera coordinate system can be calculated based on the depth map output by the network t The coordinates of P t =d uv K -1 (u,v,1) T ; Then, according to the formula P s ←TP t The three-dimensional coordinate P t is mapped to the coordinates in the source camera coordinate system. The arrows indicate the necessary conversion between homogeneous coordinates and non-homogeneous coordinates. t is the left image, then the relative position of the left camera is T = T lr , I s That is, the corresponding right image, the relative position of the right camera is T = T rl ;if I t is the right image, then the relative position of the right camera is T = T rl , I s That is, the corresponding left image, the relative position of the right camera is T = T lr Next, P s Projected to the source camera pixel coordinate system, P s The projected coordinates are calculated according to the formula (u s (u,v),v s (u,v) T ←KP sFinally, the coordinates (u s ,v s ) and use this pixel value as an approximate estimate of the pixel (u,v) in the target image.

[0058] After obtaining the synthesized target image, the reprojection error between the synthesized image and the target image is calculated. The reprojection error is filtered out for unclear pixels. The filtering method may be to use an image mask to filter out inaccurate pixels. The image mask may be a binary mask.

[0059] The reprojection error of the binocular image pair is used to characterize the L1 error and structural similarity between the synthetic image and the target image. The reprojection error satisfies

[0060]

[0061] in, Characterize the L1 error, Characterizes structural similarity; where θ takes an empirical value, is the pixel value of the synthesized image, I t is the pixel value of the target image. L1 error is the error between the corresponding pixels of the synthetic image and the target image. SSIM represents the structural similarity between the two images. The smaller the SSIM, the closer the structures of the two images. θ is the weighting coefficient of the two errors and is a hyperparameter. In practical applications, the empirical value of θ can be 0.15. When calculating the plane homography matrix and projection matrix, the objective function is constructed by using the reprojection error, and then the objective function is minimized to optimize the homography matrix or projection matrix. The reason for using the reprojection error is that it not only considers the calculation error of the homography matrix, but also the measurement error of the image points, so its accuracy will be higher.

[0062] As a possible implementation method, after the image to be measured is processed according to the monocular depth estimation model and the depth map is determined, the method further includes: scaling the depth map so that the resolution of the scaled depth map is the same as the resolution of the camera.

[0063] When the number of training times reaches the set threshold, the monocular image is input into the monocular depth estimation model for verification, and the depth map output by the last layer of the output model is output. The current output depth map is then bilinearly interpolated and enlarged to the same resolution as the camera. The enlarged depth map contains the required dense depth information.

[0064] As a possible implementation method, the objective function of the monocular depth estimation model is a minimized objective function, and the objective function is minimized by a gradient descent method.

[0065] When using binocular images to train a monocular depth estimation model, the optimization goal of the monocular depth estimation model is to minimize the objective function, and the optimization goal of the monocular depth estimation model is converted to solving the minimum value of the objective function parameter. The minimization method adopts the gradient descent method. Among them, the Adam algorithm used in this application is a kind of gradient descent algorithm, and other better gradient descent algorithms can also be selected.

[0066] The binary mask consists of H rows of elements, and the vth (0≤v≤H-1)th row of the image mask satisfies

[0067]

[0068] in, The horizontal coordinate vector of the pixels in row v of the target image, For The vector is obtained by sorting. The input image resolution of the monocular depth estimation model is W×H, 0≤v≤W-1, 0≤s≤H-1, [] represents the Iverson bracket. If the conditions in the brackets are met, it is 1, and if they are not met, it is 0. By calculating an image mask, the inaccurate pixels in the reprojection error are filtered out using the mask, and the weighted sum of the smoothness loss and the filtered reprojection error is used as the objective function, and the back propagation algorithm is used to iteratively optimize the model weight parameters. Among them, the image mask is mainly used to remove the occlusion effect.

[0069] For example, when calculating the binary mask M, the size of the mask is W×H, which is the same as the input image resolution of the monocular depth estimation model. Each element in the mask indicates whether the pixel of the target image with the corresponding coordinate is visible in the source image. "1" means that the corresponding pixel of the target image is visible in the source image, and "0" means that the corresponding pixel of the target image is not visible in the source image. Since the left and right cameras are set at the same horizontal height and have the same orientation angle, the vertical coordinates of the pixel coordinates imaged by the left and right cameras at the same point in the environment are the same, and only the horizontal coordinates are different. Therefore, the calculation of the binary mask is mainly based on the difference in the horizontal coordinates. The horizontal coordinate of the pixel in the vth row (0≤v≤H-1) of the target image can be expressed as a vector: The horizontal coordinates of the W pixels reprojected to the source image using the calculated vector are expressed as right Sort in ascending order to get the vector According to the above M v The calculation formula gets the vector of elements in the vth row.

[0070] Among them, the occlusion detection principle proposed by the present invention is as follows Figure 5 As shown, it is assumed that the left and right cameras are the target camera and the source camera respectively, and the corresponding camera focal points are Ct and C s , distant objects and near objects are imaged on the left target image, including 8 pixels from t0 to t7, of which 3 pixels from t2 to t4 are invisible in the source image due to occlusion. By calculation, the projection coordinates of the 8 pixels from t0 to t7 in the source image coordinate system are s0 to s7, where s0<s1<s2<s6<s3<s7<s4<s5, so we can get a set of sorting s0S1s2S6S3S7s4s5. Compared with s0s1s2s3s4s5s6s7, the two groups of ordered elements are compared in size, and then s0=s0, s1=s1, s2=s2, s6<S3, S3<S4, s7<s5, s4>s6, s5>s7. When the less than sign is taken, corresponding to the three points s3, s4, and s5, except s5, the other points s3 and s4 are all occluded. According to this principle, occlusion detection can be realized, and the maximum detection error of any row in the occluded area of ​​the image is only one pixel.

[0071] For example, when using the Adam gradient descent algorithm to iteratively optimize the model, the batch and the number of each batch are set according to the logarithm of the binocular image pair, and different learning rates are set for different batches of training images. When the number of training times reaches the set threshold, the training is stopped. After the training is completed, the training results are verified using monocular images. The monocular depth estimation model is saved when a better result is achieved. The objective function of the monocular depth estimation model is optimized using the gradient descent method. By improving the training method and minimizing the objective function, the network parameters that affect the model training and model output are updated and calculated, so that they approach or reach the optimal value of the network parameters of the monocular depth estimation model, making the optimization process simpler and the model prediction more accurate.

[0072] As a possible implementation method, the objective function of the monocular depth estimation model is also used to characterize the smoothness loss of the binocular image pair. The smoothness loss satisfies:

[0073]

[0074] Among them, D 0i (i=0,1,…,n-1) is the pixel value of the depth map generated by the monocular depth estimation model based on the binocular image pair, is the pixel value of the synthesized image, I t is the pixel value of the target image, 0≤u≤W-1, 0≤v≤H-1, and the input image resolution of the monocular depth estimation model is W×H. The influence of local gradients caused by bilinear interpolation in the reprojection process can be improved by adding a smoothing term.

[0075] For example, after obtaining the image mask matrix M and the reprojection error After that, the objective function is calculated according to the following formula:

[0076]

[0077] n is the number of depth maps of different resolutions output by the monocular depth estimation model based on the binocular image pair. The input image resolution of the monocular depth estimation model is W×H. M is the image mask matrix, 0≤u≤W-1, 0≤v≤H-1. For synthetic images With the target image I t The reprojection error between . The objective function after adding the smoothing term satisfies , γ is the weighting coefficient, and the empirical value is 0.001.

[0078] By filtering out inaccurate pixels in reprojection error calculation through image mask, the trained model can obtain clearer object boundaries when estimating the depth map corresponding to a single image; by taking the weighted sum of smoothness loss and filtered reprojection error as the objective function, the influence of local gradients caused by bilinear interpolation in the reprojection process can be improved, effectively improving the overall performance of the monocular depth estimation model without increasing the processing time during testing.

[0079] In one example, if Figure 6 The implementation steps shown include,

[0080] Step 1: The binocular camera collects data synchronously. In order to facilitate comparison with other self-supervised monocular depth estimation methods, we choose Eigen Split, a subset of KITTI Raw Dataset, as the training and test datasets. EigenSplit contains 45,200 training images and 697 test images. Each image corresponds to a binocular image pair, and the images contain a variety of similar resolutions such as 1242×375, 1242×375, 1241×376, etc. When the camera image resolution is scaled to 1×1, the corresponding camera intrinsic parameter matrix is:

[0081]

[0082] The relative positions of the left and right cameras are:

[0083]

[0084] Step 2: Data preprocessing. Select the encoder-decoder structure as the network structure, which is based on U-Net. In order to achieve a lightweight network structure, select ResNet18 as the encoder. The resolution of the input image of the network is 640×192. The decoder of the network outputs 4 depth maps of different resolutions. The resolution of the depth map output by the last layer is 640×192. If the hardware resources are sufficient or there is no strong requirement for real-time performance, if you want to improve the accuracy of the network model, you can also use a deeper network (such as ResNet50 as the encoder) and a larger resolution. The binocular image pair is randomly flipped left and right with a probability of 50%. The left image after flipping will be marked as the right image, and the right image will be marked as the left image; add random perturbations to the brightness, contrast and saturation of the binocular image pair with a probability of 50%, and the maximum relative perturbation is ±20%. After that, ANTIALIAS interpolation can be used to scale all images to 640×192. After scaling, the corresponding internal parameter matrix changes to

[0085]

[0086] Step 3: Calculate the reprojection error. During training, for each 640×192 target image, the decoder generates four depth maps of different resolutions. Next, all the output depth maps are upsampled to 640×192 using bilinear interpolation. Then, the reprojection coordinates (u s (u,v),v s (u,v)), the associative law of matrix multiplication can be used to reduce the number of operations, that is, (u s (u,v),v s (u,v) T ←B(d uv u,d uv v,d uv ,1) T , where B = KTK -1 ,← represents the conversion between homogeneous and non-homogeneous coordinates. In this case, just calculate the matrix B once and save it, and there is no need to repeat the calculation later. In the above formula, T is replaced by T according to whether the target image is the left image or the right image. lr or T rl Instead, at the same time, due to the need for matrix multiplication, K needs to be changed to

[0087]

[0088] In addition, when synthesizing images, the coordinates of the target image can be regarded as a 2×122880 matrix (u,v) T, and expand the depth map into a row vector d containing 122880 elements, so the reprojection process can be seen as a series of matrix operations, namely Represents the Hadamard product, which can fully utilize the GPU for acceleration. After that, bilinear interpolation is performed on the source image based on the calculated reprojected coordinates to obtain the synthesized target image. Then, the error between the target image and the synthesized image can be calculated.

[0089] Step 4: Calculate the objective function and perform iterative optimization. First, calculate the binary mask. After calculating the occlusion mask M, in order to filter out points with poor depth estimation,

[0090]

[0091] After that, we can use M to filter the reprojection error, and then perform a weighted summation with the smoothing term to get the loss function. Then, we can iteratively update the network parameters according to the loss function. During the entire training process, the batch size of each iteration is 12, and a total of 20 epochs are required. The learning rate for the first 15 epochs is 0.0001, and the learning rate for the last 5 epochs is 0.00001.

[0092] Step 5: Test model performance. Scale the single image to be tested to 640×192 and input it into the network. Then upsample the depth map output by the last layer of the network to the original image size to obtain the required depth map. Figure 5 The method of the present invention is compared with the maskless method and the Monodepth2 method based on binocular image pair training proposed by Clement Godard et al. Figure 7 It can be seen that the depth map obtained by the method of the present invention has a clearer object boundary, and the object contour is closer to the object contour in the RGB image.

[0093] In addition, the method in this paper is also evaluated and compared with the maskless method and the Monodepth2 method trained with stereo data on the test set consisting of 697 images of the KITTI dataset. Monodepth2 is the model with the highest performance among all self-supervised monocular depth estimation models so far. To ensure the fairness of the evaluation, the three methods all use the same network structure, and the training and test sets used are also the same, and the image size of the network input is also consistent. In addition, the evaluation is also compared with the SuperDepth model proposed by Sudeep Pillai et al. It should be noted that the resolution of the network input image of SuperDepth is 1024×382, and all four methods are trained only with stereo image pairs. The evaluation uses the general evaluation benchmark for self-supervised monocular depth estimation, which was proposed by David Eigen. The evaluation indicators include 7 indicators such as Abs Rel, Sq Rel, RMSE, and RMSE log. The measurement results are shown in Table 1. The bold data indicates that the method performs best on this indicator, and the underlined data indicates that the data of this indicator ranks second. According to the measurement results, it can be seen that adding a mask can effectively improve the performance, and most of the performance indicators of the method of the present invention exceed SuperDepth and Monodepth2, and the overall performance is the best.

[0094] Table 1

[0095]

[0096] All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0097] In the case of using the corresponding integrated unit, Figure 8 FIG. 5 is a block diagram showing a structure of an image processing device 500 provided in an embodiment of the present invention. The image processing device 500 may be Figure 2 The image processing device 100 shown may also be applied to Figure 2 The chip of the image processing device 100 is shown.

[0098] like Figure 8 As shown, the image processing device 500 includes: a communication unit 501 and a processing unit 502. Optionally, the image processing device 500 may also include a storage unit 503 for storing program codes and data of the image processing device 500.

[0099] In one example, if Figure 8 As shown, the processing unit 502 is used to support the image processing device 500 to perform the above-mentioned embodiment. Figure 3 The image processing device 100 shown executes steps 301 to 302.

[0100] Among them, the processing unit 502 can be a graphics processor or controller, for example, it can be a central graphics processor (GPU), a general graphics processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. The graphics processor can also be a combination that implements computing functions, such as a combination of one or more micro-graphics processors, a combination of DSP and micro-graphics processors, and the like. The communication unit 501 can be a transceiver, a transceiver circuit or a communication interface, and the like. The storage unit 503 can be a memory.

[0101] When the processing unit 502 is a graphics processor, the communication unit 501 is a transceiver, and the storage unit 503 is a memory, the image processing device 500 involved in the embodiment of the present invention can be Figure 4 FIG. 1 is a schematic diagram of the hardware structure of the image processing device 100 .

[0102] Fig. 9 FIG. 1 is a schematic diagram showing the hardware structure of the device 100 provided in an embodiment of the present invention. Fig. 9 As shown, the apparatus 100 includes a graphics processor 110 and a communication interface 130 .

[0103] like Fig. 9 As shown, the graphics processor 110 may be a general-purpose central processing unit (GPU), a micro graphics processor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present invention. The communication interface may be one or more. The communication interface 130 may use any transceiver-like device for communicating with other devices or communication networks.

[0104] like Fig. 9 As shown, the apparatus 100 may further include a communication line 140. The communication line 140 may include a path for transmitting information between the components.

[0105] Optional, such as Fig. 9As shown, the device 100 may further include a memory 120. The memory 120 is used to store computer-executable instructions for executing the solution of the present invention, and is controlled to execute by the graphics processor 110. The graphics processor 110 is used to execute the computer-executable instructions stored in the memory 120, thereby implementing the method provided by the embodiment of the present invention.

[0106] like Fig. 9 As shown, the memory 120 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 120 may exist independently and be connected to the graphics processor 110 via the communication line 140. The memory 120 may also be integrated with the graphics processor.

[0107] Optionally, the computer-executable instructions in the embodiment of the present invention may also be referred to as application program codes, which is not specifically limited in the embodiment of the present invention.

[0108] In a specific implementation, as an example, Fig. 9 As shown, the graphics processor 110 may include one or more GPUs, such as Fig. 9 GPU0 and GPU1 in .

[0109] In a specific implementation, as an example, Fig. 9 As shown, the image processing device 100 may include multiple graphics processors, such as Fig. 9 The graphics processor 110 and the graphics processor 150 in FIG. Each of these graphics processors may be a single-core graphics processor or a multi-core graphics processor.

[0110] Fig.10 600 is a schematic diagram of the structure of the chip 600 provided in an embodiment of the present invention. Fig.10 As shown, the chip 600 includes one or more (including two) graphics processors 610 and a communication interface 620 .

[0111] Optional, such as Fig.10 As shown, the chip 600 further includes a memory 630, which may include a read-only memory 630 and a random access memory 630, and provides operation instructions and data to the graphics processor 610. A portion of the memory may also include a non-volatile random access memory (NVRAM).

[0112] In some embodiments, Fig.10 As shown, the memory 630 stores the following elements, execution modules or data structures, or their subsets, or their extended sets.

[0113] In the embodiment of the present invention, Fig.10 As shown, the corresponding operation is performed by calling the operation instruction stored in the memory 630 (the operation instruction may be stored in the operating system).

[0114] like Fig.10 As shown, the graphics processor 610 controls the processing operations of any one of the image processing devices 100 , and the graphics processor 610 may also be referred to as a central processing unit (GPU).

[0115] like Fig.10 As shown, the memory 630 may include a read-only memory and a random access memory, and provide instructions and data to the graphics processor 610. A portion of the memory 630 may also include an NVRAM. For example, in an application, the memory 630, the communication interface 620, and the memory 630 are coupled together through a bus system, wherein the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, in Fig.10 Various buses are labeled as bus system 640 .

[0116] like Fig.10As shown, the method disclosed in the above embodiment of the present invention can be applied to a graphics processor 610, or implemented by a graphics processor 610. The graphics processor 610 may be an integrated circuit chip 600, which has the ability to process signals. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the graphics processor 610 or by instructions in the form of software. The above graphics processor 610 can be a general graphics processor, a digital signal graphics processor (digital signal processing, DSP), an ASIC, a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general graphics processor can be a micro graphics processor or the graphics processor can also be any conventional graphics processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as being executed by a hardware decoding graphics processor, or can be executed by a combination of hardware and software modules in a decoding graphics processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 630 , and the graphics processor 610 reads the information in the memory 630 and completes the steps of the above method in combination with its hardware.

[0117] The embodiment of the present invention further provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed, the functions performed by the image processing device 100 in the above embodiment are implemented.

[0118] On the one hand, a chip 600 is provided, which is applied to a material property testing device 100. The chip 600 includes at least one graphics processor 610 and a communication interface 620. The communication interface 620 is coupled to at least one graphics processor 610. The graphics processor 610 is used to run instructions to implement the functions performed by the image processing device 100 in the above embodiment.

[0119] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on the computer, the process or function of the embodiment of the present invention is executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device or other programmable device. The computer program or instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instruction can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by the computer or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as a solid state drive (SSD).

[0120] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. A single graphics processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0121] Although the present invention has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present invention. Accordingly, this specification and the accompanying drawings are merely exemplary illustrations of the present invention as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present invention. Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such modifications and variations if they fall within the scope of the claims of the present invention and their equivalents.

Claims

1. A method for generating image depth information, characterized in that: include: Acquire the image to be tested; The image to be tested is processed according to a monocular depth estimation model to determine depth information; wherein the monocular depth estimation model is a model trained based on a binocular image pair, and an objective function of the monocular depth estimation model is determined by at least a corrected reprojection error of the binocular image pair, and the corrected reprojection error is a reprojection error that filters out inaccurate pixels; The objective function of the monocular depth estimation model is to satisfy: Wherein, n is the number of depth maps of different resolutions output by the monocular depth estimation model according to the binocular image pair, and the width of the input picture resolution of the monocular depth estimation model is W and the length is H; M is a matrix of an image mask, the image mask includes H rows of elements, and the vth row of elements of the image mask satisfies: Among them, M v is the v-th row element of the image mask, is the horizontal coordinate vector of the vth row of pixels after the target image is reprojected onto the source image, For Sort to get a vector; is the reprojection error, and the reprojection error satisfies: in, Characterize the L1 error, Characterizes structural similarity; where θ takes an empirical value, is the pixel value of the synthesized image, i is the i-th synthesized image, I t is the pixel value of the target image; L s (I t ,D 0i ) is a smoothness loss, and the smoothness loss satisfies: Among them, D 0i is the pixel value of the depth map generated by the monocular depth estimation model according to the binocular image pair, i=0,1,…,n-1, 0≤u≤W-1, 0≤v≤H-1, γ is a weighting coefficient, which takes an empirical value.

2. The method according to claim 1, characterized in that: The objective function of the monocular depth estimation model is a minimized objective function, and the minimization method of the objective function is a gradient descent method.

3. The method according to claim 1, characterized in that The binocular image pair includes a target image and a source image, and before filtering out the inaccurate pixels, a reprojection error of the binocular image pair is determined by an L1 error and a structural similarity between a synthesized image and the target image, and the synthesized image is determined by a depth map of the target image and the source image; is the reprojection error, and the reprojection error satisfies: in, Characterizing the L1 error, Characterize the structural similarity; where θ takes an empirical value, is the pixel value of the synthesized image, i is the i-th synthesized image, I t is the pixel value of the target image.

4. The method according to claim 1, characterized in that: The corrected reprojection error is a reprojection error obtained by filtering out inaccurate pixels using an image mask; the image mask includes H rows of elements, and the vth row of elements of the image mask satisfies: Among them, M v is the v-th row element of the image mask, is the horizontal coordinate vector of the vth row of pixels after the target image is reprojected onto the source image, 0≤v≤H-1, For The vector is obtained by sorting, and the width of the input image resolution of the monocular depth estimation model is W and the length is H.

5. The method according to claim 1, characterized in that The objective function of the monocular depth estimation model is determined by the corrected reprojection error of the binocular image pair and the smoothness loss of the binocular image pair.

6. The method according to claim 1, characterized in that After the image to be tested is processed according to the monocular depth estimation model to determine the depth map, the method further includes: The depth map is scaled so that a resolution of the scaled depth map is the same as a resolution of the camera.

7. An image processing device, characterized in that: include: A graphics processor and a communication interface coupled to the graphics processor; the graphics processor is used to run a computer program or instruction to implement the image depth information generation method as described in any one of claims 1 to 6.

8. A computer storage medium, characterized in that: The computer storage medium stores instructions, and when the instructions are executed, the image depth information generating method according to any one of claims 1 to 6 is implemented.

Images