Binocular depth estimation method, apparatus, embedded device and readable storage medium

Abstract

The application relates to the field of image processing, and provides a binocular depth estimation method and device, an embedded device and a readable storage medium, the method comprising the following steps: acquiring a binocular image and performing feature extraction on the binocular image to obtain left and right feature mappings; performing disparity construction by using the left and right feature mappings to obtain a reduced dimension disparity cost volume; performing attention feature learning on the disparity cost volume to obtain an attention feature vector, and performing feature weighting on the disparity cost volume by using the attention feature vector to obtain a weighted cost volume; performing disparity regression on the weighted cost volume based on two-dimensional convolution to obtain a predicted disparity map; and performing disparity depth conversion on the predicted disparity map to obtain a depth map of the binocular image. The scheme solves the problems of large calculation amount and large memory of the prior art, and can realize real-time operation on a low-power embedded platform.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a binocular depth estimation method and device, an embedded device and a readable storage medium. BACKGROUND

[0002] Binocular depth estimation refers to finding matching pixel points on acquired left and right view images, then calculating corresponding parallax, and further combining known binocular camera information to calculate real physical distance (i.e. depth information). Traditional binocular depth estimation algorithms are mostly local or global search algorithms, which calculate the matching degree of left and right view image pixels by constructing a cost function, and select the pixel with the minimum cost in the search range as the matching point.

[0003] However, these traditional algorithms have some problems in actual use. For example, the traditional method constructs a five-dimensional parallax cost space, so it needs to rely on a large number of three-dimensional convolutions to realize depth estimation, which has the problems of large memory occupation and large calculation amount, especially for low-power embedded devices, which cannot be transplanted and used, and cannot perform real-time depth estimation. In addition, because there are too many matching candidates, the accuracy is not high. SUMMARY

[0004] Therefore, the embodiments of the present application provide a binocular depth estimation method, device, embedded device and readable storage medium.

[0005] In a first aspect, the embodiments of the present application provide a binocular depth estimation method, comprising:

[0006] acquiring binocular images and performing feature extraction on the binocular images to obtain left and right feature mappings;

[0007] performing parallax construction using the left and right feature mappings to obtain a reduced-dimensional parallax cost volume;

[0008] performing attention feature learning on the parallax cost volume to obtain an attention feature vector, and performing feature weighting on the parallax cost volume using the attention feature vector to obtain a weighted cost volume;

[0009] performing parallax regression on the weighted cost volume based on two-dimensional convolution to obtain a predicted parallax map;

[0010] performing parallax depth conversion on the predicted parallax map to obtain a depth map of the binocular images.

[0011] In some embodiments, the performing reduced-dimensional parallax construction using the left and right feature mappings to obtain a reduced-dimensional parallax cost volume comprises:

[0012] The left and right feature maps of the initial channel dimension are respectively subjected to channel dimension reduction to obtain left and right feature maps of a first channel dimension;

[0013] The right feature map of the first channel dimension is moved in a horizontal direction by a pixel step and is subjected to channel dimension splicing with the left feature map of the first channel dimension to obtain a disparity feature map of a second channel dimension;

[0014] The left and right feature maps of the first channel dimension and the disparity feature map of the second channel dimension are subjected to channel dimension splicing to obtain a disparity feature map of an initial channel dimension;

[0015] The disparity feature map of the initial channel dimension is subjected to channel dimension reduction to obtain a disparity feature map with a channel dimension of 1 as a reduced disparity cost volume.

[0016] In some embodiments, the disparity feature map of the initial channel dimension is subjected to channel dimension reduction to obtain a disparity feature map with a channel dimension of 1, including:

[0017] The left and right feature maps of the initial channel dimension are subjected to group correlation calculation to obtain a group correlation feature map;

[0018] The left and right feature maps of the first channel dimension and the group correlation feature map are subjected to convolution processing to obtain a disparity feature map with a channel dimension of 1.

[0019] In some embodiments, the attention feature learning on the disparity cost volume includes:

[0020] An attention feature vector is extracted from the disparity cost volume by using a trained convolution network, and the trained convolution network is obtained by:

[0021] A loss of attention feature learning on a training sample by a constructed convolution network is calculated by using a smooth L1 loss function, and parameters in the convolution network are iteratively updated until an attention feature vector output by the convolution network after parameter updating satisfies a preset convergence condition.

[0022] In some embodiments, the disparity regression on the weighted cost volume based on a two-dimensional convolution includes:

[0023] A two-dimensional convolution network is trained, and a disparity feature is extracted from the weighted cost volume by using the trained two-dimensional convolution network, and the disparity feature is regularized by using a normalization exponential function to obtain a probability of each pixel at different disparity levels, and then a disparity prediction value of each pixel is obtained by weighting and calculating the probability and an index of the corresponding disparity level, and a continuous predicted disparity map is generated.

[0024] In some embodiments, during the training of the two-dimensional convolutional network, the cost volume is learned by the two-dimensional convolutional network for disparity prediction, a predicted disparity map is obtained, and a loss value between the predicted disparity map and a real disparity map is calculated using a smooth L1 loss function to update parameters of the two-dimensional convolutional network.

[0025] In some embodiments, the disparity depth conversion on the predicted disparity map to obtain a depth map of the binocular image comprises:

[0026] Based on a focal length and a baseline distance of the binocular camera, the predicted disparity map is converted between a camera plane and a real world to obtain a depth map corresponding to the binocular image pair in the real world.

[0027] In some embodiments, before the feature extraction on the binocular image, the binocular depth estimation method further comprises:

[0028] The obtained binocular image is preprocessed, and the preprocessed binocular image will be used for the feature extraction operation; wherein the preprocessing comprises performing epipolar alignment on the binocular image, and then performing pixel normalization on the aligned binocular image.

[0029] In a second aspect, the embodiments of the present application provide a binocular depth estimation device, comprising:

[0030] A feature extraction module is configured to obtain a binocular image and perform feature extraction on the binocular image to obtain left and right feature maps.

[0031] A cost volume construction module is configured to perform disparity construction using the left and right feature maps to obtain a reduced dimension disparity cost volume.

[0032] An attention module is configured to perform attention feature learning on the disparity cost volume to obtain an attention feature vector, and perform feature weighting on the disparity cost volume using the attention feature vector to obtain a weighted cost volume.

[0033] A disparity regression module is configured to perform disparity regression on the weighted cost volume based on a two-dimensional convolution to obtain a predicted disparity map.

[0034] A conversion module is configured to perform disparity depth conversion on the predicted disparity map to obtain a depth map of the binocular image.

[0035] In a third aspect, the embodiments of the present application provide an embedded device, which comprises a binocular camera, a processor and a memory, the binocular camera is configured to obtain a binocular image, the memory stores a computer program, and the processor is configured to execute the computer program to implement the binocular depth estimation method.

[0036] In a fourth aspect, the embodiments of the present application provide a readable storage medium storing a computer program, which, when executed on a processor, implements the binocular depth estimation method.

[0037] The embodiments of the present application have the following beneficial effects:

[0038] The binocular depth estimation method of the embodiments of the present application obtains left and right feature mappings by first obtaining a binocular image and performing feature extraction on the binocular image; then, disparity construction is performed using the left and right feature mappings to obtain a reduced dimension disparity cost volume; next, attention feature learning is performed on the disparity cost volume to obtain an attention feature vector, and the attention feature vector is used to perform feature weighting on the disparity cost volume to obtain a weighted cost volume; finally, two-dimensional convolution is used to perform disparity regression on the weighted cost volume to obtain a predicted disparity map, and disparity depth conversion is performed on the predicted disparity map to obtain a depth map of the binocular image. Since the reduced dimension processing is performed when the disparity cost volume is constructed, the cost volume becomes 1 in the channel dimension, which can on one hand realize the supplement operation in the disparity dimension, and on the other hand, the subsequent disparity regression model can use a two-dimensional convolution structure instead of a three-dimensional convolution structure in the traditional scheme, the model calculation amount is small, the occupied memory is small, and the above-mentioned operation can be supported by embedded devices, so that the entire scheme can be transplanted to low-power embedded devices. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as limiting the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0040] Figure 1 The main architecture diagram of the binocular depth estimation method of the embodiments of the present application is shown;

[0041] Figure 2 The flowchart of the binocular depth estimation method of the embodiments of the present application is shown;

[0042] Figure 3 The epipolar constraint schematic diagram of the binocular depth estimation method of the embodiments of the present application is shown;

[0043] Figure 4 The epipolar alignment schematic diagram of the binocular depth estimation method of the embodiments of the present application is shown;

[0044] Figure 5 The disparity cost volume construction flowchart of the binocular depth estimation method of the embodiments of the present application is shown;

[0045] Figure 6 A schematic diagram of a dimensionality reduction cost volume construction of the binocular depth estimation method of the embodiments of the present application is shown;

[0046] Figure 7 A structural schematic diagram of a binocular depth estimation device of the embodiments of the present application is shown. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application.

[0048] The components of the embodiments of the present application generally described and illustrated in the accompanying drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0049] Hereinafter, the terms "include", "have", and their conjugates used in the various embodiments of the present application are only intended to denote that specific features, numbers, steps, operations, elements, components, or combinations thereof are present, and should not be understood as excluding the presence or addition of one or more other features, numbers, steps, operations, elements, components, or combinations thereof. In addition, the terms "first", "second", "third", and the like are only used to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0050] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the present application belong. The terms (such as those defined in commonly used dictionaries) will be interpreted as having a meaning that is the same as the contextual meaning in the relevant technical field and will not be interpreted in an idealized or overly formal sense, unless clearly defined in the various embodiments of the present application.

[0051] Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0052] Disparity is the position difference between corresponding pixel points in horizontal direction of left and right two images, and the target of binocular depth information estimation is to calculate the disparity of each pixel point on the reference image, therefore, the accuracy of depth will be improved with the accuracy of disparity prediction. Generally, the traditional binocular depth estimation mainly includes the following four steps: matching cost calculation, cost aggregation, disparity calculation and disparity optimization. However, in the existing scheme, for the step of matching cost calculation, a disparity cost volume with a total dimension of 5 dimensions is usually constructed, which can be described as a cost space with a total dimension of [B, C, D, H, W], wherein B represents Batch, C represents Channel, D represents disparity, H represents Height, and W represents Width. It can be understood that, since the more the total number of dimensions is, the more the memory occupied by the disparity cost space is, and the more complex the subsequent processing steps of cost aggregation, disparity calculation and disparity optimization are, and correspondingly, the larger the calculation amount is, which is not suitable for embedded devices due to insufficient computing power and limited memory, so that the traditional scheme cannot be implemented on embedded devices, resulting in that binocular depth estimation is restricted on such devices.

[0053] Based on this, the present application proposes a lightweight binocular depth estimation method. After feature extraction is performed on binocular images, as shown in Figure 1 , dimension reduction processing is adopted when constructing the disparity cost volume, and at the same time, in order to ensure the accuracy after dimension reduction, an attention mechanism is also combined to weight the disparity features by using an additional supervision signal, and only a two-dimensional convolution model is needed to realize disparity prediction in the disparity regression process. At the same time, due to the reduction of model complexity, the embedded device can perform format conversion on the constructed model file in the format supported by the device, so as to be transplanted into the device for use. The method can realize real-time operation on a low-power embedded platform, effectively solving the problem that the existing scheme cannot be implemented on embedded devices.

[0054] The binocular depth estimation method will be described below in combination with some specific embodiments.

[0055] Figure 2 A flowchart of the binocular depth estimation method of the embodiments of the present application is shown. Exemplarily, the binocular depth estimation method is mainly used in embedded devices, that is, the embedded device is used to perform the following steps:

[0056] S110, acquiring binocular images and performing feature extraction on the binocular images to obtain left and right feature maps.

[0057] Exemplarily, a binocular camera can be installed on the embedded device to acquire binocular images (also referred to as left and right images or left and right eye images) in real time, and then feature extraction is performed based on the acquired binocular images to obtain feature mappings of the left and right images respectively. In the feature extraction, a pre-trained neural network can be used, and the specific structure is not limited, for example, a lightweight neural network such as a MobileNet series can be used as a feature extraction model.

[0058] Considering that if the original binocular images are directly used for feature extraction and depth estimation, there is no prior constraint between the binocular images, and in the pixel matching, each pixel of the left image needs to be searched in the whole image space of the right image, which greatly reduces the matching efficiency. Therefore, the binocular images are preprocessed before the feature extraction operation to greatly improve the efficiency of subsequent disparity calculation.

[0059] As a preferred scheme, before the feature extraction of the binocular images, the method further includes preprocessing the acquired binocular images, and it can be understood that the preprocessed binocular images will be used for subsequent feature extraction operation. The preprocessing can include but is not limited to the following: performing epipolar alignment on the acquired original binocular images, and then performing pixel normalization on the aligned binocular images.

[0060] Exemplarily, in the epipolar alignment operation, as shown in Figure 3 the epipolar alignment operation is completed. Therefore, the corresponding points in the left and right images after the epipolar alignment only have differences in column numbers (i.e., the disparity), which can ensure that the objects in the binocular images are on the same horizontal line. It can be understood that after the epipolar alignment, the X axis of the camera coordinate system is parallel to the baseline, and the Z axis is perpendicular to the image plane, as shown in Figure 4 If the disparity can be obtained, the required depth information can be calculated according to the following formula based on the principle of triangle similarity.

[0061] In addition, after epipolar alignment, pixel normalization is performed on the binocular images for the convenience of image feature extraction. Specifically, the pixel distribution of the left and right images is normalized from the initial 0-255 to 0-1, and then each value in the RGB channel image is subtracted by the average value of the channel, and the result is divided by the standard deviation of the channel, thereby obtaining the normalized binocular image. Finally, the normalized binocular image is input into the feature extraction model to obtain the left and right feature maps.

[0062] In S120, disparity construction is performed using the left and right feature maps to obtain a reduced dimension disparity cost volume.

[0063] The cost volume is used to measure the similarity of two image blocks in the left and right binocular images in binocular matching to determine whether they match. It is worth noting that, unlike the traditional cost volume construction method, in the construction of the disparity cost volume, the embodiment not only performs a dimension reduction operation in the channel dimension by multiplying the multi-layer convolution and the feature map, but also proposes a cyclic addition and append method based on the dimension reduction operation, that is, only the new space in the disparity dimension is continuously increased to obtain a 4-dimensional disparity cost volume. Since the append operation in a single dimension can be supported by the current embedded device, it is convenient to implement on an embedded platform.

[0064] Exemplarily, the embodiment realizes the dimension reduction operation by combining the multi-layer convolution structure with the group-wise correlation calculation, thereby constructing a cost space with a channel number of 1 (C=1), which reduces the memory occupation to 1 / C, and in the subsequent disparity regression process, two-dimensional convolution can also be used for calculation. Compared with the traditional scheme of using three-dimensional convolution, the calculation amount can be reduced to 1 / 3 of the traditional scheme, which is convenient for transplantation for embedded devices with low computing power.

[0065] In one embodiment, in combination with Figure 5 and 6 Step S120 includes the following sub-steps:

[0066] In S121, channel dimension reduction is performed on the left and right feature maps with an initial channel dimension to obtain left and right feature maps with a first channel dimension.

[0067] For example, the first channel dimension can be set to C / 4, etc. Specifically, a common convolution layer structure can be used to reduce the left and right feature maps with an initial channel dimension of C to C / 4. The convolution layer for channel dimension reduction can use some common dimension reduction convolution structures, which are not limited here.

[0068] S122, moving the right feature map of the first channel dimension in the horizontal direction by a pixel step, and performing channel dimension splicing with the left feature map of the first channel dimension to obtain a second channel dimension disparity feature map.

[0069] After that, the reduced dimension feature map is spliced in the channel dimension, thereby obtaining the second channel dimension disparity feature map. It can be understood that after the previous polar alignment and other preprocessing operations, the left and right images are already on the same horizontal line. When moving in the horizontal direction (i.e. the width W direction of the image), the two same objects with disparity will overlap in the channel, thereby artificially constructing different disparities.

[0070] Exemplarily, the reduced dimension right feature map can be moved in the horizontal direction by a step of one pixel each time, wherein the disparity corresponds to plus one each time. After moving one pixel, the spliced feature map of the second channel dimension corresponding to the disparity can be obtained by splicing with the left feature map of the first channel dimension. Finally, all spliced feature maps are combined to obtain the above-mentioned disparity feature map. It can be understood that the second channel dimension is equal to the sum of two first channel dimensions. Still taking the above-mentioned C / 4 as an example, the second channel number is equal to C / 2.

[0071] S123, performing channel dimension splicing on the left and right feature maps of the first channel dimension and the disparity feature map of the second channel dimension to obtain a disparity feature map of an initial channel dimension.

[0072] After that, the left and right feature maps and the disparity feature map after the dimension reduction are spliced again, that is, a disparity feature map with a channel number of C is finally obtained.

[0073] S124, performing channel dimension reduction on the disparity feature map of the initial channel dimension to obtain a disparity feature map with a channel dimension of 1 as a reduced disparity cost volume.

[0074] Finally, in order to construct the cost volume, the embodiment divides the initial left and right feature maps into a plurality of groups along the channel dimension, and then cross-correlates the i-th left feature map group with the corresponding i-th feature map group at all disparity levels, thereby obtaining a group correlation graph, that is, the group correlation is used to provide a better similarity measurement of the left and right images.

[0075] Specifically, the left and right feature maps of the initial channel dimension can be calculated to obtain a group correlation feature map; then, the left and right feature maps of the first channel dimension and the group correlation feature map are convolved to obtain a disparity feature map with a channel dimension of 1.

[0076] Exemplarily, the disparity cost volume can be expressed by the following expression:

[0077]

[0078]

[0079] In the formula, C(d, x, y) represents a disparity cost volume, represents a convolution operation, and c is the number of channels; respectively represent left feature maps and right feature maps with a channel dimension of c / 4, (x, y) represents the coordinates of a pixel point in a left image, and (x-d, y) represents the coordinates of a pixel point in a right image with a disparity of d; G wc represents group correlation calculation, and N c / N g represents the number of channels of each feature map group, where Ng is the number of divided groups; respectively represent left feature map groups and right feature map groups obtained by dividing left feature maps and right feature maps with a channel dimension of c into g groups, respectively.

[0080] It can be understood that, by the above convolution and dimension reduction operations, a cost space with a channel number of C=1, a disparity number of D, and a total dimension of [B, 1, D, H, W] can be constructed, where, since C=1, the total dimension of the cost volume can be considered as only [B, D, H, W], that is, the memory occupation in the channel dimension is reduced by 1 / C. In addition, when constructing the cost space, the embedded device can use the supported augmentation operation to continuously increase new space only in the disparity dimension.

[0081] S130, attention feature learning is performed on the disparity cost volume to obtain an attention feature vector, and the attention feature vector is used for feature weighting on the disparity cost volume to obtain a weighted cost volume.

[0082] In this embodiment, after the disparity cost volume is constructed, an attention mechanism is combined to improve the subsequent matching accuracy. Illustratively, a trained convolutional network can be used to perform attention feature extraction on the disparity cost volume to obtain an attention feature vector; then, the attention feature vector is used for feature weighting on the disparity cost volume to obtain an optimized disparity cost volume.

[0083] wherein the attention feature vector is used for feature weighting on the disparity cost volume, that is, the two are subjected to an inner product operation, which can be specifically expressed as: C' 代 =C 代 *w attention , in the formula, C' 代 represents the weighted disparity cost volume, C 代 represents the unweighted disparity cost volume, and w attentiondenotes an attention feature vector. It can be understood that, since the channel dimension of the cost volume is reduced to 1, the dimension of the attention feature vector is [B, D, H, W].

[0084] For the trained convolutional network described above, the training method can be implemented by using a common network model training process. For example, a Smooth L1 loss function or the like can be used to calculate the loss of the constructed convolutional network when performing attention feature learning on the training sample, so as to iteratively update the parameters in the convolutional network, until the attention feature vector output by the convolutional network after parameter updating meets a preset convergence condition, and the training is stopped. For example, the preset convergence condition can be that the error between the supervision signal for supervised learning and the attention feature vector learned and output by the convolutional network does not exceed the allowed error range, and of course other conditions can also be used, which are not limited here.

[0085] In S140, disparity regression is performed on the weighted cost volume based on two-dimensional convolution, to obtain a predicted disparity map.

[0086] After obtaining the disparity cost volume weighted by the attention feature, the embodiment performs disparity regression on the disparity cost volume, to obtain a disparity map. It is worth noting that, since the channel dimension of the disparity cost volume has been reduced to 1, when performing disparity regression, only a two-dimensional convolutional network can be used to extract the disparity feature. Compared with the traditional scheme which needs to rely on a three-dimensional convolutional network, the amount of calculation is obviously reduced. In addition, since the model files of existing three-dimensional convolutional networks are mostly written in formats such as python, and the embedded device cannot support the format conversion of such complex model files, that is, the model file cannot be directly transplanted. In the embodiment, since the complexity of the convolutional network is reduced to two dimensions, the model file of the two-dimensional convolutional network can be converted from the python format to the onnx format, and then converted to a format supported by the embedded device through the development tool SDK of the embedded device, so as to realize the use of the model file on the embedded device.

[0087] Exemplarily, the trained two-dimensional convolutional network can be used to extract the disparity feature from the weighted cost volume, and a normalized exponential function (i.e. Softmax function) can be used to regularize the disparity feature, to obtain the probability of each pixel at different disparity levels. Then, the probabilities and the indexes of the corresponding disparity levels are weighted and calculated, to obtain the disparity prediction value of each pixel, and then a continuous predicted disparity map is generated. It can be understood that the calculation of the disparity is a non-differentiable problem, and therefore the Softargmin method which is differentiable can be used to regress the disparity value of the pixel from the disparity cost volume.

[0088] Specifically, the calculation of the predicted disparity can be described by the following expression:

[0089]

[0090] wherein, represents the disparity prediction value of a single pixel, and Dmax represents the total number of disparities; represents the probability of the current disparity level d in all possible disparities j C j .

[0091] In an embodiment, in the training process of the two-dimensional convolutional network described above, the cost volume can be learned by the two-dimensional convolutional network for disparity prediction, to obtain a predicted disparity map, and a Smooth L1 loss function is used to calculate the loss value between the predicted disparity map and the real disparity map, so as to update the parameters of the two-dimensional convolutional network until the final output of the disparity prediction result can meet the corresponding accuracy requirements.

[0092] Specifically, the loss function L described above can be described as:

[0093]

[0094] wherein, g i represents the i-th real disparity map, represents the i-th predicted disparity map, and N is the total number of pixels, represents the normalization processing on the objects in the parentheses.

[0095] S150, the predicted disparity map is converted into a depth map of the binocular image.

[0096] Exemplarily, when the disparity depth conversion is performed, the predicted disparity map can be converted between the camera plane and the real world based on the focal length and the baseline distance of the binocular camera, to obtain the depth map corresponding to the binocular image pair in the real world.

[0097] In combination with the foregoing Figure 4 , assuming that P is a point in the real world, f is the camera focal length, O L O R is the baseline of the plane, b is the baseline distance, and Z is the depth distance of P to the baseline, so:

[0098] wherein, X L and X R represent the projections P L and P R of the P point on the left and right camera planes, respectively, and the straight line distance from the left camera plane, so that the depth information Z in the real world can be converted.

[0099] The binocular depth estimation method of the embodiment is to preprocess the obtained binocular image, then extract features, and then adopt dimension reduction processing when constructing the disparity cost volume, so that the cost volume becomes 1 in the channel dimension; in addition, the attention mechanism is combined to weight the disparity features by using an additional supervision signal, and finally a two-dimensional convolution model is adopted in the subsequent disparity regression process to realize disparity prediction, so that the model has small calculation amount and small memory occupation, and the above operations can be supported by an embedded device, so that the method can realize real-time operation on a low-power embedded platform, effectively solving the problem that the existing scheme cannot be landed in an embedded device.

[0100] Figure 7 A structural schematic diagram of a binocular depth estimation device 100 of an embodiment of the application is shown. Exemplarily, the binocular depth estimation device 100 includes:

[0101] A feature extraction module 110 is configured to acquire binocular images and extract features from the binocular images to obtain left and right feature maps.

[0102] A cost volume construction module 120 is configured to construct a disparity based on the left and right feature maps to obtain a reduced dimension disparity cost volume.

[0103] An attention module 130 is configured to learn attention features from the disparity cost volume to obtain an attention feature vector, and weight the disparity cost volume based on the attention feature vector to obtain a weighted cost volume.

[0104] A disparity regression module 140 is configured to regress the weighted cost volume based on a two-dimensional convolution to obtain a predicted disparity map.

[0105] A conversion module 150 is configured to convert the predicted disparity map into a depth map of the binocular image.

[0106] It can be understood that the binocular depth estimation device of the embodiment corresponds to the binocular depth estimation method of the above-mentioned embodiment, and the optional items about the method in the above-mentioned embodiment are also applicable to the embodiment, so they will not be described here again.

[0107] The application also provides an embedded device, for example, the embedded device can be but is not limited to a smart robot, a communication terminal, a monitoring device, and other terminal devices based on an embedded platform, such as a binocular camera, which is not limited here.

[0108] Exemplarily, the embedded device comprises a binocular camera, a processor and a memory, wherein the binocular camera is configured to acquire binocular images, the memory stores a computer program, and the processor executes the computer program to cause the embedded device to perform the functions of the above-mentioned binocular depth estimation method or each module of the above-mentioned binocular depth estimation device.

[0109] The processor can be an integrated circuit chip with a signal processing capability. The processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), and a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., and can implement or execute the disclosed methods, steps, and logic block diagrams in the embodiments of the present application.

[0110] The memory can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory is configured to store a computer program. After receiving an execution instruction, the processor can execute the computer program accordingly.

[0111] The present application also provides a readable storage medium for storing the computer program used in the above-mentioned embedded device.

[0112] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can also be implemented by other means. The apparatus embodiments described above are only illustrative, for example, the flowcharts and structural diagrams in the drawings show the possible implementation architecture, function and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that in alternative implementation, the functions noted in the blocks can also occur in different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the structural diagram and / or flowchart, and the combination of blocks in the structural diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0113] In addition, each functional module or unit in the embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0114] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0115] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application.

Claims

1. A binocular depth estimation method, characterized in that, The method comprises the following steps: acquiring binocular images and performing feature extraction on the binocular images to obtain left and right feature maps; performing disparity construction using the left and right feature maps to obtain a disparity cost volume based on channel dimension reduction; performing attention feature learning on the disparity cost volume to obtain an attention feature vector, and performing feature weighting on the disparity cost volume using the attention feature vector to obtain a weighted cost volume; performing disparity regression on the weighted cost volume based on two-dimensional convolution to obtain a predicted disparity map; performing disparity depth conversion on the predicted disparity map to obtain a depth map of the binocular images; wherein the disparity regression on the weighted cost volume based on two-dimensional convolution to obtain a predicted disparity map comprises: performing disparity feature extraction on the weighted cost volume using a trained two-dimensional convolution network, and performing regularization processing on the disparity features using a normalized exponential function to obtain the probability of each pixel at different disparity levels, and then performing weighted sum and calculation according to the probability and the index of the corresponding disparity level to obtain the disparity prediction value of each pixel, and further generating a continuous predicted disparity map.

2. The binocular depth estimation method of claim 1, wherein, The method comprises the following steps: performing channel dimension reduction on the left and right feature maps of the initial channel dimension to obtain left and right feature maps of the first channel dimension; moving the right feature map of the first channel dimension in the horizontal direction according to the pixel step, and performing channel dimension splicing with the left feature map of the first channel dimension to obtain a disparity feature map of the second channel dimension; performing channel dimension splicing on the left and right feature maps of the first channel dimension and the disparity feature map of the second channel dimension to obtain a disparity feature map of the initial channel dimension; performing channel dimension reduction on the disparity feature map of the initial channel dimension to obtain a disparity feature map with a channel dimension of 1 as a reduced disparity cost volume.

3. The binocular depth estimation method of claim 2, wherein, The method comprises the following steps: performing group correlation calculation on the left and right feature maps of the initial channel dimension to obtain a group correlation feature map; performing convolution processing on the left and right feature maps of the first channel dimension and the group correlation feature map to obtain a disparity feature map with a channel dimension of 1.

4. The binocular depth estimation method of claim 1, wherein, The method comprises the following steps: performing attention feature extraction on the disparity cost volume using a trained convolution network to obtain an attention feature vector; wherein the trained convolution network is obtained by: calculating the loss of attention feature learning of the constructed convolution network on the training sample using a smooth L1 loss function, and iteratively updating the parameters in the convolution network until the attention feature vector output by the convolution network after parameter updating meets the preset convergence condition.

5. The binocular depth estimation method of claim 1, wherein, In the training process of the two-dimensional convolutional network, the cost volume is learned by the two-dimensional convolutional network for disparity prediction to obtain a predicted disparity map, and a smooth L1 loss function is used to calculate a loss value between the predicted disparity map and a real disparity map to update parameters of the two-dimensional convolutional network.

6. The binocular depth estimation method of claim 1, wherein, The disparity depth conversion on the predicted disparity map is to obtain a depth map of the binocular image, comprising: Based on the focal length and baseline distance of the binocular camera, the predicted disparity map is converted between the camera plane and the real world to obtain a depth map corresponding to the binocular image in the real world.

7. The binocular depth estimation method according to any of claims 1 to 6, wherein, Before the feature extraction on the binocular image, further comprising: The obtained binocular image is preprocessed, and the preprocessed binocular image will be used for the feature extraction operation; wherein the preprocessing includes polar alignment of the binocular image, and then pixel normalization processing on the aligned binocular image.

8. A binocular depth estimation apparatus, characterized by, Comprising: The feature extraction module is configured to obtain a binocular image and perform feature extraction on the binocular image to obtain left and right feature mappings; The cost volume construction module is configured to perform disparity construction on the left and right feature mappings to obtain a disparity cost volume based on channel dimension reduction; The attention module is configured to perform attention feature learning on the disparity cost volume to obtain an attention feature vector, and perform feature weighting on the disparity cost volume using the attention feature vector to obtain a weighted cost volume; The disparity regression module is configured to perform disparity regression on the weighted cost volume based on a two-dimensional convolution to obtain a predicted disparity map, comprising: performing disparity feature extraction on the weighted cost volume using a trained two-dimensional convolutional network, and performing regularization processing on the disparity features using a normalized exponential function to obtain the probability of each pixel at different disparity levels, and then performing weighted sum and calculation according to the probability and the index of the corresponding disparity level to obtain the disparity prediction value of each pixel, thereby generating a continuous predicted disparity map; The conversion module is configured to perform disparity depth conversion on the predicted disparity map to obtain a depth map of the binocular image.

9. An embedded device, characterized by The embedded device comprises a binocular camera, a processor and a memory, the binocular camera is configured to obtain a binocular image, the memory stores a computer program, and the processor is configured to execute the computer program to implement the binocular depth estimation method of any one of claims 1-7.

10. A readable storage medium, characterized by, It stores a computer program, and the computer program implements the binocular depth estimation method according to any one of claims 1-7 when executed on a processor.

Images