Disparity estimation system, method, electronic device and computer readable storage medium

Abstract

A disparity estimation system, method, electronic device and computer-readable storage medium are provided. The disparity estimation system includes: a feature extraction network configured to extract features from each image in an image pair and output the extracted image features to a disparity generation network; and the disparity generation network configured to perform cascaded multi-level disparity processing according to the extracted image features to obtain multiple disparity maps with successively increasing sizes, wherein the input of the first level of disparity processing in the multi-level disparity processing includes multiple image features with sizes corresponding to the disparity processing at this level; and the input of each level of disparity processing in the multi-level disparity processing except the first level of disparity processing includes: one or more image features with sizes corresponding to the disparity processing at this level, and a disparity map generated by the previous level of disparity processing.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a disparity estimation system, method, electronic device, and computer-readable storage medium. Background Art

[0002] In the related art, computer vision technology can be used to obtain the disparity between each pair of matching pixel points in two images of the same scene from different perspectives, obtain a disparity map, and obtain the depth information of the scene based on the disparity map, wherein the depth information can be used in various fields such as three-dimensional reconstruction, autonomous driving, and obstacle detection. For example, the method of obtaining disparity using computer vision technology can include a local area matching method, a global optimization method, a semi-global method, and a method based on a convolutional neural network, etc.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] According to one aspect of the present disclosure, a disparity estimation system is provided, comprising: a feature extraction network, configured to extract features from each image in an image pair and output the extracted image features to a disparity generation network; and the disparity generation network, configured to perform cascaded multi-level disparity processing based on the extracted image features to obtain multiple disparity maps of successively increasing sizes, wherein the input of a first level of disparity processing in the multi-level disparity processing includes multiple image features having sizes corresponding to the disparity processing of this level; the input of each level of disparity processing in the multi-level disparity processing except the first level of disparity processing includes: one or more image features having sizes corresponding to the disparity processing of this level, and a disparity map generated by the previous level of disparity processing.

[0005] According to another aspect of the present disclosure, a disparity estimation method is provided, comprising: extracting features from each image in an image pair; and performing cascaded multi-level disparity processing based on the extracted image features to obtain a plurality of disparity maps of successively increasing sizes, wherein the input of a first level of disparity processing in the multi-level disparity processing includes a plurality of image features having sizes corresponding to the disparity processing of this level; the input of each level of disparity processing in the multi-level disparity processing except the first level of disparity processing includes: one or more image features having sizes corresponding to the disparity processing of this level, and a disparity map generated by the previous level of disparity processing.

[0006] According to another aspect of the present disclosure, an electronic device is provided, including: a processor; and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the method described in the present disclosure.

[0007] According to another aspect of the present disclosure, a computer-readable storage medium storing a program is provided. The program includes instructions, which, when executed by a processor of an electronic device, cause the electronic device to execute the method described in the present disclosure.

[0008] Further features and advantages of the present disclosure will become apparent from the exemplary embodiments described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0010] Figure 1 is a block diagram showing a structure of a disparity estimation system according to an exemplary embodiment of the present disclosure;

[0011] Figure 2 is a schematic diagram showing basic structural features of an image according to an exemplary embodiment of the present disclosure;

[0012] Figure 3 is a schematic diagram showing semantic features of an image according to an exemplary embodiment of the present disclosure;

[0013] Figure 4 is a schematic diagram showing edge features of an image according to an exemplary embodiment of the present disclosure;

[0014] Figure 5 is a block diagram showing a possible overall structure of a disparity estimation system according to an exemplary embodiment of the present disclosure;

[0015] Figure 6 is a block diagram showing another possible overall structure of a disparity estimation system according to an exemplary embodiment of the present disclosure;

[0016] Fig. 7A and Figure 7B are schematic diagrams showing a reference image based on which network training is performed and a corresponding true disparity map according to an exemplary embodiment of the present disclosure;

[0017] Figure 8 FIG. 2 is a diagram showing a method for using a trained disparity estimation system according to an exemplary embodiment of the present disclosure. Fig. 7AA schematic diagram of a plurality of disparity maps whose sizes increase from right to left in sequence, obtained by performing cascaded multi-level disparity processing on the reference image shown;

[0018] Fig. 9 is a flowchart illustrating a disparity estimation method according to an exemplary embodiment of the present disclosure;

[0019] Fig.10 is a flowchart illustrating a multi-level disparity process in a disparity estimation method according to an exemplary embodiment of the present disclosure;

[0020] Fig.11 is a block diagram showing the structure of an exemplary computing device that can be applied to the exemplary embodiments of the present disclosure. DETAILED DESCRIPTION

[0021] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0022] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0023] In the related art, computer vision technology can be used to obtain the disparity between each pair of matching pixel points in two images of the same scene from different perspectives, obtain a disparity map, and obtain the depth information of the scene based on the disparity map, wherein the depth information can be used in various fields such as three-dimensional reconstruction, autonomous driving, and obstacle detection. For example, the method of obtaining disparity using computer vision technology can include a local area matching method, a global optimization method, a semi-global method, and a method based on a neural network such as a convolutional neural network, etc.

[0024] The local area matching method mainly includes the steps of matching cost calculation, cost aggregation, disparity calculation and disparity optimization. It has high speed and low energy consumption. However, its algorithm effect is related to the algorithm parameters (such as the size of the matching window, etc.), and it is difficult to meet the needs of complex scenes. Compared with the local area matching method, the global optimization method has better matching accuracy. It makes assumptions about the smoothness term and turns the disparity calculation problem of stereo matching into an energy optimization problem. Most global optimization methods skip the cost aggregation step. By considering the matching cost and smoothness term, an energy function is proposed for the global point, and the disparity is obtained by minimizing the energy function. However, compared with the local area matching method, the global optimization method has a larger amount of calculation and higher energy consumption. The semi-global method can balance the matching accuracy and operation speed to a certain extent. Unlike the global algorithm that optimizes the global point, it divides the energy function of each point into multiple directional paths. It only needs to evaluate each path, and then add the values ​​of all paths to get the energy of the point. The evaluation of each path can be done by dynamic programming. However, compared with the local area matching method, the semi-global method has higher calculation amount and energy consumption. Methods based on neural networks such as CNN (Convolutional Neural Network) can obtain a larger perception domain by constructing a disparity network, and have better disparity prediction capabilities in texture-free areas of the image. However, its computational complexity is related to the parameters of neural networks such as CNN and the image size. The more complex the network parameters and the larger the image size, the greater the memory consumption and the lower the running speed.

[0025] The present disclosure provides a new disparity estimation system, which can perform cascaded multi-level disparity processing based on the image features of each image in the extracted image pair to obtain multiple disparity maps with increasing sizes, wherein the input of the first level disparity processing in the multi-level disparity processing may include multiple image features with sizes corresponding to the disparity processing of this level; the input of each level disparity processing except the first level disparity processing in the multi-level disparity processing may include: one or more image features with sizes corresponding to the disparity processing of this level, and the disparity map generated by the previous level disparity processing. In other words, by performing cascaded multi-level disparity processing on the extracted image features, wherein the input of each level disparity processing may include image features with sizes corresponding to the disparity processing of this level, multiple disparity maps of different sizes can be obtained at one time for use by multiple target devices with different performance or different accuracy requirements, thereby meeting the requirements of different target devices for accuracy and speed, and also improving the flexibility and applicability of the disparity estimation system. The exemplary embodiments of the disparity estimation system of the present disclosure will be further described below in conjunction with the accompanying drawings.

[0026] Figure 1is a block diagram showing a disparity estimation system according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the disparity estimation system 100 may, for example, include: a feature extraction network 200, configured to extract features from each image in an image pair and output the extracted image features to a disparity generation network 300; and the disparity generation network 300, configured to perform cascaded multi-level disparity processing based on the extracted image features to obtain a plurality of disparity maps of successively increasing sizes, wherein the input of a first level of disparity processing in the multi-level disparity processing includes a plurality of image features having sizes corresponding to the disparity processing at that level; the input of each level of disparity processing in the multi-level disparity processing except the first level of disparity processing includes: one or more image features having sizes corresponding to the disparity processing at that level, and a disparity map generated by the previous level of disparity processing.

[0027] according to Figure 1 The disparity estimation system shown in the structural block diagram can perform cascaded multi-level disparity processing based on the image features of each image in the extracted image pair to obtain multiple disparity maps with successively increasing sizes, wherein the input of each level of disparity processing may include image features with a size corresponding to the level of disparity processing. Thus, multiple disparity maps of different sizes can be obtained at one time for use by multiple target devices with different performance or different accuracy requirements, thereby meeting the accuracy and speed requirements of different target devices, and also improving the flexibility and applicability of the disparity estimation system.

[0028] In the present disclosure, the image pair may be an image pair for the same scene acquired by a multi-camera. The sizes of the images in the image pair are consistent, but the corresponding viewing angles are different. Of course, the image pair may also be an image pair that meets the requirements and is acquired by other means (such as acquired from other third-party devices). In addition, each image in the image pair may be a grayscale image or a color image.

[0029] In the present disclosure, the multi-eye camera refers to a camera that is configured with two, three or even more cameras and is capable of capturing static or dynamic images. It can cover scenes of different viewing angles or ranges through the configured multiple cameras to enhance the camera's ability to detect objects in the scene. Taking a binocular camera configured with two cameras (e.g., a left camera and a right camera) as an example, for any scene, the binocular camera can obtain two images of the scene with the same size but different corresponding shooting viewing angles (e.g., a left-eye image and a right-eye image) through the configured two cameras, wherein the image pair formed by the two images can be used to determine the displacement (e.g., horizontal displacement) of the object in the scene between the corresponding pixel points in the two images, i.e., parallax, so as to determine depth information such as the distance of the object.

[0030] In addition, in the present disclosure, the disparity estimation system 100 and the multi-eye camera may be independent of each other. In other words, the disparity estimation system 100 may extract features from each image in the image pair for the same scene collected by the multi-eye camera through the included feature extraction network 200, and perform cascaded multi-level disparity processing on the extracted image features through the included disparity generation network 300 to obtain multiple disparity maps with increasing sizes. As an alternative, the multi-eye camera may also be used as a part of the disparity estimation system 100. In other words, in addition to the feature extraction network 200 and the disparity generation network 300, the disparity estimation system 100 may also include the multi-eye camera.

[0031] According to some embodiments, the image features of each image in the image pair extracted by the feature extraction network 200 of the disparity estimation system 100 may include at least one or more of the following features: basic structure features, semantic features, edge features, texture features, color features, object shape features, or features based on the image itself.

[0032] Figure 2 The three images (a), (b) and (c) (for example, grayscale images, or color images) are schematic diagrams showing basic structural features of an image that may be extracted according to an exemplary embodiment of the present disclosure. Figure 2 It can be seen that the basic structural features may refer to features used to reflect various fine structures of an image.

[0033] Figure 3 Four images (a), (b), (c) and (d) (for example, grayscale images, or color images) are schematic diagrams showing semantic features of an image that may be extracted according to an exemplary embodiment of the present disclosure. Figure 3 It can be seen that the semantic feature may refer to a feature capable of distinguishing different objects or objects of different categories in an image. In addition, the accuracy of disparity determination in ambiguous regions (eg, large flat regions) of an image may be improved based on the semantic feature.

[0034] Figure 4 Two images (a) and (b) (eg, grayscale images, or color images) are schematic diagrams showing edge features of an image that may be extracted according to an exemplary embodiment of the present disclosure. Figure 4 It can be seen that the edge feature may refer to a feature that can reflect boundary information of an object or region in an image.

[0035] In addition, although not shown, the texture feature, color feature, and object shape feature may refer to features that can be used to reflect the texture, color, and shape of the object contained in the image, respectively. The feature based on the image itself may refer to the image itself, or may be an image obtained by upsampling or downsampling the image itself with a certain coefficient or ratio. The coefficient or ratio of the upsampling or downsampling may be, for example, 2, 3, or other values ​​greater than 1.

[0036] According to some embodiments, in the present disclosure, in addition to the features based on the image itself, each other image feature can be extracted by the corresponding feature extraction sub-network on the corresponding image to improve the efficiency of image feature extraction and thus improve the efficiency of disparity estimation. In addition, in order to improve the accuracy of disparity estimation, feature extraction can be performed on the image from at least three different dimensions: basic structural features, semantic features, and edge features.

[0037] For example, according to some embodiments, Figure 5 or Figure 6 As shown, Figure 5 and Figure 6 A possible overall structural block diagram of a disparity estimation system 100 according to an exemplary embodiment of the present disclosure is shown, wherein the feature extraction network 200 may include a plurality of feature extraction subnetworks for respectively extracting different features of an image, and the plurality of feature extraction subnetworks may include at least a basic structural feature subnetwork 201 for extracting basic structural features of an image, a semantic feature subnetwork 202 for extracting semantic features of an image, and an edge feature subnetwork 203 for extracting edge features of an image.

[0038] According to some embodiments, the basic structure feature sub-network 201 may adopt any network that can be used to extract the basic structure features of an image, such as VGG (very deep convolutional networks for large scale image recognition) or ResNet (Residual Network). The semantic feature sub-network 202 may adopt any network that can be used to extract the semantic features of an image, such as DeepLabV3+ (encoder-decoder with atrous separable convolution for semantic image segmentation). The edge feature sub-network 203 may adopt any network that can be used to extract the edge features of an image, such as a HED (holistically-nested edge detection) network. According to some embodiments, the HED network may adopt VGG as the backbone network, and when the edge feature sub-network 203 adopts the HED network, the basic structure feature sub-network 201 may adopt the same VGG network as the edge feature sub-network 203, so as to achieve the effect of simplifying the structure of the feature extraction network.

[0039] According to some embodiments, the feature extraction network 200 or each feature extraction subnetwork included in the feature extraction network 200 may be an extraction network pre-trained based on a training sample set, so that the efficiency of image feature extraction can be improved, thereby improving the efficiency of disparity estimation. Of course, according to actual needs, the feature extraction network 200 or each feature extraction subnetwork included in the feature extraction network 200 may also be obtained by real-time training based on a training sample set, or may also be obtained by real-time or timed optimization of a pre-trained extraction network based on an updated training sample set, so as to improve the accuracy of features extracted by the feature extraction network.

[0040] According to some embodiments, the training process of the feature extraction network 200 or each feature extraction subnetwork included in the feature extraction network 200 may adopt supervised training or unsupervised training, which can be flexibly selected according to actual needs. Supervised training usually uses existing training samples (such as labeled data) to learn the mapping from input to output, and then applies this mapping relationship to unknown data to achieve the purpose of classification or regression. Supervised training algorithms may include, for example, logistic regression algorithms, SVM (Support Vector Machine) algorithms, decision tree algorithms, and the like. The difference between unsupervised training and supervised training is that unsupervised training does not require training samples, but directly models unlabeled data to find out the rules therein. Its typical algorithms may include, for example, clustering algorithms, random forests algorithms, and the like.

[0041] According to some embodiments, as described above, the input of the first level of disparity processing in the multi-level disparity processing may include multiple image features with sizes corresponding to the disparity processing of this level; the input of each level of disparity processing except the first level of disparity processing in the multi-level disparity processing may include: one or more image features with sizes corresponding to the disparity processing of this level. Taking the example that the multiple disparity maps obtained by the disparity generation network 300 are N disparity maps of increasing sizes in sequence, the image features extracted by the feature extraction network 200 may include image features of N sizes, where N is a positive integer not less than 2. At least a portion of the image features of each size may be used to help generate a disparity map of a corresponding size to improve the accuracy of the multiple disparity maps obtained by the disparity estimation system. According to some embodiments, the value of N may be 4 (e.g., Figure 5 or Figure 6 As shown in FIG. 1 , of course, it can also be set to 2, 3, 5 or other values ​​according to actual needs. In addition, the value of N is not the larger the better, but a suitable value can be selected on the premise of balancing the accuracy requirements of the target device and the speed of the disparity estimation system.

[0042] In the present disclosure, the size of each image (including each image in the image pair and the disparity map, etc.) may refer to the size of a single channel of each image, which may be represented by the height and width of the image, for example, may be represented as H x W, where H represents the height of the image, W represents the width of the image, and the units of both may be pixels. Of course, this is only an example, and the size of the image may also be represented by one or more parameters that can reflect the number of pixels, data volume, storage volume, or clarity of the image. In addition, it should be noted that for a grayscale image, the number of channels is 1, and for a color image, since it may have three color channels of R, G, and B, the number of channels may be 3, that is, the actual size of the color image may be represented as H x W x 3. In addition, in the present disclosure, the size of each image in the image pair (i.e., the size of the original image that has not been processed by downsampling and / or upsampling) may be determined based on parameters such as the size and number of pixels of the sensor of the multi-eye camera used to capture the image pair.

[0043] In the present disclosure, the size corresponding to each level of disparity processing may refer to a size consistent with the size of the disparity map required for each level of disparity processing. In addition, the size of the image feature may refer to the size of a single channel of the image constituted by the image feature itself, or the size of the extracted image based on which the image feature of the required size is extracted, and the extracted image may refer to each image in the image pair itself, or may be an image obtained by upsampling or downsampling each image itself by a certain coefficient or ratio. For example, taking the size of the image in the image pair as H x W (which may be referred to as the full size) as an example, the full-size image features extracted from the image may be image features obtained by extracting features from the image itself; the image features of the size of 1 / 2H x 1 / 2W (which may be referred to as the 1 / 2 size) extracted from the image may be image features obtained by downsampling the image by 2 times to obtain a 1 / 2-size image, and extracting features from the 1 / 2-size image.

[0044] According to some embodiments, the input of each level of disparity processing except the first level of disparity processing in the multi-level disparity processing may include, in addition to one or more image features having a size corresponding to the level of disparity processing, a disparity map generated by the previous level of disparity processing. In other words, based on one or more image features having corresponding sizes among the extracted image features, the disparity map generated by the first level of disparity processing may be optimized step by step to obtain a disparity map of corresponding size. As a result, the accuracy of the disparity map obtained subsequently can be gradually improved without having to calculate the disparity map from scratch for each accuracy, so that the overall generation efficiency of multiple disparity maps can be improved.

[0045] According to some embodiments, the image feature of the smallest size among the N sizes of image features extracted by the feature extraction network 200 may, for example, include at least one image feature of the first image in the image pair and at least one image feature of the second image, and each non-minimum size of the image features of the N sizes may, for example, include at least one image feature of the first image and / or at least one image feature of the second image in the image pair.

[0046] For example, Figure 5 or Figure 6 As shown, the image features of the smallest size among the N sizes of image features extracted by the feature extraction network 200 may include basic structural features, semantic features, edge features of the first image (e.g., the left image) in the image pair and basic structural features of the second image (e.g., the right image) in the image pair. Each non-smallest size image feature among the N sizes of image features extracted by the feature extraction network 200 may include edge features of the first image in the image pair or features based on the first image itself.

[0047] According to some embodiments, each image feature of each image extracted by the feature extraction network 200 may have one or more sizes, and the number of the multiple sizes may be less than or equal to N. For example, Figure 5 or Figure 6 As shown, the value of N can be 4, the edge features of the first image extracted by the feature extraction network 200 and the features based on the first image itself can have two sizes respectively, the basic structural features and semantic features extracted from the first image can have one size respectively, and the basic structural features extracted from the second image can have one size. In addition, Figure 5 or Figure 6 This is just an example. In addition to the one or two sizes shown, each image feature of each image extracted by the feature extraction network 200 may also have more sizes. For example, taking the value of N as 4 as an example, the edge features of the first image extracted by the feature extraction network 200 may also have three or four sizes, which is not limited.

[0048] According to some embodiments, after the feature extraction network 200 extracts the image features of each image in the image pair, it can store (e.g., cache) it in a storage device or storage medium for subsequent reading and use. In addition, before the feature extraction network 200 extracts the image features of each image in the image pair, it can also perform epipolar correction on the images in the image pair so that the images in the image pair have disparity only in one direction (e.g., horizontal direction or vertical direction). As a result, the disparity search range of the image can be limited to one direction, thereby improving the efficiency of subsequent feature extraction and disparity generation. As an alternative, the epipolar correction operation of the images in the image pair can also be performed by a multi-eye camera or other third-party device. For example, after the multi-eye camera captures the image pair, the images in the image pair can be epipolar corrected, and the epipolar corrected image pair can be sent to the disparity estimation system. Alternatively, after the multi-eye camera captures the image pair, it can send it to other third-party devices, and the other third-party devices perform epipolar correction on the images in the image pair and send the epipolar corrected image pair to the disparity estimation system.

[0049] According to some embodiments, the disparity map with the largest size among the multiple disparity maps obtained by the disparity generation network 300 may be consistent with the size of each image in the image pair (i.e., the original size of each image). Thus, through cascaded multi-level disparity processing, at least a disparity map with a relatively high accuracy and a disparity map with other accuracy that corresponds to the size of each image in the image pair can be obtained, which can better meet the requirements of high-performance target devices for the accuracy of the disparity map generated by the disparity estimation system on the basis of improving the flexibility and applicability of the disparity estimation system. As an alternative, the size of each disparity map in the multiple disparity maps may also be smaller than the size of each image in the image pair.

[0050] According to some embodiments, the height and width of the latter disparity map in any two adjacent disparity maps in the multiple disparity maps may be twice the height and width of the previous disparity map, respectively. For example, taking the multiple disparity maps as 4 and the size of the last disparity map in the multiple disparity maps as H x W (which may be consistent with the size of each image in the image pair), the sizes of the other disparity maps in the multiple disparity maps before it may be 1 / 2H x 1 / 2W (if H x W can be called full size, then 1 / 2H x 1 / 2W can be called 1 / 2 size), 1 / 4H x 1 / 4W (can be called 1 / 4 size), and 1 / 8H x 1 / 8W (can be called 1 / 8 size). In other words, in the present disclosure, the value 2 is used as the scaling step size of the height and width of adjacent disparity maps (or the coefficient or ratio of upsampling or downsampling of adjacent disparity maps). As an alternative, the height and width of the latter disparity map in any two adjacent disparity maps among the multiple disparity maps can also be 3 times, 4 times or other positive integer multiples greater than 1 of the height and width of the previous disparity map, respectively, and appropriate values ​​can be selected according to the actual required accuracy.

[0051] According to some embodiments, as mentioned above, the image features extracted by the feature extraction network 200 may include image features of N sizes, where N is a positive integer not less than 2. Accordingly, the disparity generation network may be configured to generate an initial disparity map having the minimum size according to at least a portion of the image features of the minimum size among the image features of the N sizes in the first level of disparity processing of the multi-level disparity processing; and to perform disparity optimization processing on the disparity map generated by the previous level of disparity processing according to at least a portion of the image features of the corresponding size among the image features of the N sizes in each subsequent level of disparity processing of the multi-level disparity processing, so as to generate an optimized disparity map having the corresponding size, wherein the multiple disparity maps may at least include each optimized disparity map.

[0052] According to some embodiments, the multi-level disparity processing may include N+1 level disparity processing. The disparity generation network may be configured to, in N levels of disparity processing other than the first level of disparity processing, sequentially perform disparity optimization processing on the disparity map generated by the previous level of disparity processing based on at least a portion of the image features with corresponding sizes in the image features of the N sizes in order from small to large, to obtain N optimized disparity maps with increasing sizes, and use the N optimized disparity maps as the multiple disparity maps, wherein the sizes of the N optimized disparity maps correspond to the N sizes respectively.

[0053] For example, Figure 5As shown, the multi-level parallax processing may include 4+1 level parallax processing. The extracted image features may include image features of 4 sizes, wherein: Figure 5 The extracted image features of the four sizes are 1 / 8 size (1 / 8H x 1 / 8W), 1 / 4 size (1 / 4H x 1 / 4W), 1 / 2 size (1 / 2H x 1 / 2W) and full size (H x W, the full size may refer to a size consistent with the size of the original image in the image pair) as an example for schematic description. The disparity generation network may be configured to generate an initial disparity map with the minimum size (i.e., 1 / 8 size) in the first-level disparity processing of the multi-level disparity processing according to at least a portion of the image features of the minimum size (i.e., 1 / 8 size) among the image features of the four sizes (e.g., the extracted basic structural features of the first image of 1 / 8 size, the semantic features of the first image of 1 / 8 size, the edge features of the second image of 1 / 8 size, and part or all of the basic structural features of the second image of 1 / 8 size). In the four levels of disparity processing other than the first level disparity processing, in order from small to large, based on at least a part of the image features with corresponding sizes among the image features of the four sizes (for example, based on part or all of the extracted edge features of 1 / 8 size of the first image, part or all of the extracted edge features of 1 / 4 size of the first image, extracted features based on the first image itself at 1 / 2 size, and extracted features based on the first image itself at full size), the disparity optimization processing is performed on the disparity map generated by the previous level of disparity processing to obtain four optimized disparity maps of increasing size in sequence (for example, an optimized disparity map with 1 / 8 size, an optimized disparity map with 1 / 4 size, an optimized disparity map with 1 / 2 size, and an optimized disparity map with full size) and the four optimized disparity maps are used as the multiple disparity maps.

[0054] From the above description, it can be seen that in this embodiment, the multiple disparity maps obtained by the disparity estimation system 100 may not include the initial disparity map generated by the first-level disparity processing in the multi-level disparity processing, but include optimized disparity maps after successive optimization of the initial disparity map generated by the first-level disparity processing. Therefore, the accuracy of the multiple disparity maps obtained by the disparity estimation system can be improved.

[0055] According to some other embodiments, the multi-level disparity processing may include N-level disparity processing. The disparity generation network may be configured to, in N-1-level disparity processing other than the first-level disparity processing, sequentially perform disparity optimization processing on the disparity map generated by the previous level of disparity processing based on at least a portion of the image features with corresponding sizes in N-1 non-minimum-sized image features among the N-sized image features, in order of size from small to large, to obtain N-1 optimized disparity maps with increasing sizes, and use the initial disparity map and the N-1 optimized disparity map as the multiple disparity maps, wherein the sizes of the initial disparity map and the N-1 optimized disparity map correspond to the N sizes, respectively.

[0056] For example, Figure 6 As shown, the multi-level parallax processing may include 4-level parallax processing. The extracted image features may include image features of 4 sizes, wherein: Figure 6 The extracted image features of the four sizes are 1 / 8 size (1 / 8H x 1 / 8W), 1 / 4 size (1 / 4H x 1 / 4W), 1 / 2 size (1 / 2H x 1 / 2W) and full size (H x W, the full size may refer to a size consistent with the size of each image in the image pair) as an example for schematic description. The disparity generation network may be configured to generate an initial disparity map with the minimum size (i.e., 1 / 8 size) in the first-level disparity processing of the multi-level disparity processing according to at least a portion of the image features of the minimum size (i.e., 1 / 8 size) among the image features of the four sizes (e.g., the extracted basic structural features of the first image of 1 / 8 size, the semantic features of the first image of 1 / 8 size, the edge features of the second image of 1 / 8 size, and part or all of the basic structural features of the second image of 1 / 8 size). In the other three levels of disparity processing except the first level disparity processing, in order of size from small to large, disparity optimization processing is performed on the disparity map generated by the previous level of disparity processing based on at least a part of the image features with corresponding sizes among the other three non-minimum-sized image features (for example, based on part or all of the edge features of 1 / 4 size extracted from the first image, features based on the first image itself of 1 / 2 size extracted, and features based on the first image itself of full size extracted), to obtain three optimized disparity maps of increasing size in sequence (for example, an optimized disparity map with 1 / 4 size, an optimized disparity map with 1 / 2 size, and an optimized disparity map with full size are obtained), and the initial disparity map and the three optimized disparity maps are used as the multiple disparity maps.

[0057] From the above description, it can be seen that in this embodiment, the multiple disparity maps obtained by the disparity estimation system 100 may also include an initial disparity map generated by the first level disparity processing in the multi-level disparity processing to improve the processing efficiency of the disparity estimation system.

[0058] According to some embodiments, the disparity generation network 300 may be configured to, in each level of disparity processing except the first level of disparity processing in the multi-level disparity processing, perform residual calculation processing on the disparity map generated by the previous level of disparity processing based on at least a portion of the image features having a corresponding size to obtain a residual map having the corresponding size, and combine the residual map having the corresponding size with the disparity map generated by the previous level of disparity processing to obtain an optimized disparity map having the corresponding size.

[0059] For example, Figure 5 As shown, in the 1 / 8-size disparity processing corresponding to the other 4 levels of disparity processing except the first level of disparity processing, a 1 / 8-size first residual map can be calculated based on part or all of the 1 / 8-size edge features of the extracted first image and the initial disparity map (1 / 8-size) generated by the previous level of disparity processing, and the first residual map is combined (e.g., added) with the initial disparity map to obtain a 1 / 8-size first optimized disparity map as the output of the disparity processing of this level. In the next level of disparity processing corresponding to 1 / 4-size, a 1 / 4-size second residual map can be calculated based on part or all of the 1 / 4-size edge features of the extracted first image and the first optimized disparity map (1 / 8-size) generated by the previous level of disparity processing, and the second residual map is combined with the first optimized disparity map (e.g., the second residual map is added to the 1 / 4-size upsampled version of the first optimized disparity map) to obtain a 1 / 4-size second optimized disparity map as the output of the disparity processing of this level, and so on. More specific examples will be discussed later.

[0060] According to some embodiments, the disparity generation network 300 may also be configured to, in each level of disparity processing except the first level of disparity processing in the multi-level disparity processing, before performing disparity optimization processing on the disparity map generated by the previous level of disparity processing, in response to the size of the disparity map generated by the previous level of disparity processing being smaller than the size corresponding to the current level of disparity processing, upsample the disparity map generated by the previous level of disparity processing to a size corresponding to the current level of disparity processing. The algorithm used for the upsampling may include, for example, a nearest neighbor interpolation algorithm, a bilinear interpolation algorithm, a deconvolution algorithm, and the like. In this way, the disparity map targeted by each level of disparity optimization processing may be a disparity map having a size corresponding to the level of disparity processing.

[0061] For example, Figure 5As shown, in the 1 / 8-size disparity processing corresponding to the other 4 levels of disparity processing except the first level of disparity processing, since the size of the initial disparity map generated by the previous level of disparity processing is not smaller than that of the current level of disparity processing, the initial disparity map generated by the previous level of disparity processing can be directly subjected to disparity optimization processing based on part or all of the 1 / 8-size edge features of the extracted first image to obtain a 1 / 8-size first optimized disparity map. In the next level of disparity processing corresponding to 1 / 4-size, the 1 / 8-size first optimized disparity map generated by the previous level of disparity processing can first be upsampled to a 1 / 4-size corresponding to the current level of disparity processing, and then, based on part or all of the 1 / 4-size edge features of the extracted first image, the upsampled 1 / 4-size first optimized disparity map can be subjected to disparity optimization processing to obtain a 1 / 4-size second optimized disparity map. For example, as mentioned above, in the next level of disparity processing corresponding to 1 / 4 size, a 1 / 4 size residual map can be calculated based on part or all of the 1 / 4 size edge features of the extracted first image and the upsampled 1 / 4 size first optimized disparity map, and the 1 / 4 size residual map is added to the upsampled 1 / 4 size first optimized disparity map to obtain a 1 / 4 size second optimized disparity map, and so on.

[0062] According to some embodiments, the image features based on which different optimized disparity maps are generated may be image features of the same type or image features of different types; and / or, the image features based on which different optimized disparity maps are generated may be image features of the same image or different images in the image pair. Figure 5 As shown, the image features based on which the first two different optimized disparity maps are generated can be the same type of image features (e.g., edge features) of the same image (e.g., the first image) in the image pair; the image features based on which the middle two different optimized disparity maps are generated can be different types of image features (e.g., one is the edge feature of the first image, the other is the feature based on the image itself) of the same image (e.g., the first image), etc. Therefore, by flexibly selecting the image features based on which each level of disparity optimization is based, the flexibility and applicability of the disparity estimation system can be further improved.

[0063] According to some embodiments, the image features based on which each optimized disparity map is generated may include, for example, edge features of at least one image in the image pair, and / or features of the image itself based on at least one image in the image pair.

[0064] For example, Figure 5As shown, in the first two levels of relatively small-sized disparity optimization processing corresponding to the multiple disparity processing except the first-level disparity processing, an optimized disparity map of corresponding size can be generated based on the edge features of the first image; in the second two levels of relatively large-sized disparity optimization processing corresponding to the last level, the features based on the first image itself can be used to replace the edge features to perform disparity optimization processing on the disparity map generated by the previous level of disparity processing, so as to reduce the amount of calculation required for large-sized image feature extraction and improve the processing efficiency of the disparity estimation system. Of course, Figure 5 This is just an example. For each disparity optimization process, the image features based on which the corresponding optimized disparity map is generated can be edge features or features based on the image itself, or a combination of the two, or a combination of one or more other extracted image features, and so on.

[0065] According to some embodiments, the features of the image itself based on at least one image in the image pair may include, for example, the image itself of the at least one image, or an image obtained by downsampling the image itself of the at least one image according to the size of the optimized disparity map to be generated. The downsampling process may, for example, be: for an image of size H x W, if the downsampling coefficient or ratio used is K, then one point may be selected from every K points in each row and column of the original image to form an image. As mentioned above, the downsampling coefficient or ratio may be 2, 3 or other values ​​greater than 1. Of course, this is only an example, and downsampling may also be implemented in other ways, such as averaging K points.

[0066] by Figure 5 As shown, in the latter two levels of disparity optimization processing corresponding to 1 / 2 size and full size in the multiple disparity processing, a 1 / 2 size first image obtained by downsampling the first image itself based on a downsampling factor 2 and the first image itself can be used to replace the edge features of the corresponding sizes to perform disparity optimization processing on the disparity map generated by the previous level of disparity processing, so as to reduce the amount of calculation required for extracting large-size image features and improve the processing efficiency of the disparity estimation system.

[0067] According to some embodiments, Figure 5 or Figure 6 As shown, the disparity generation network 300 may include an initial disparity generation subnetwork 301 and at least one disparity optimization subnetwork 302. The initial disparity generation subnetwork 301 and each disparity optimization subnetwork 302 in the at least one disparity optimization subnetwork 302 are cascaded in sequence. The initial disparity generation subnetwork 301 is configured to perform a first level disparity processing, and the at least one disparity optimization subnetwork 302 is configured to perform various levels of disparity processing except the first level disparity processing.

[0068] It can be seen from the above embodiments that the image feature extraction of the disparity estimation system 100 and each level of disparity processing in the multi-level disparity processing can be implemented by a corresponding sub-network.

[0069] For example, the following will be Figure 5 Taking 200 as an example, the overall working process of the disparity estimation system 100 including multiple feature extraction subnetworks (such as the basic structure feature subnetwork 201, the semantic feature subnetwork 202 and the edge feature subnetwork 203), the initial disparity generation subnetwork 301 and multiple (such as four) disparity optimization subnetworks 302 is schematically illustrated.

[0070] like Figure 5 It can be seen that for the first image I1 and the second image I2 (the size of each image may be H x W) in the image pair input to the disparity estimation system 100, the disparity estimation system 100 may extract image features of the size required for subsequent multi-level disparity processing based on the multiple feature extraction subnetworks in the feature extraction network 200 included therein. For example, the basic structure features of the first image I1 with a size of 1 / 8 (1 / 8H x 1 / 8W) and the basic structure features of the second image I2 with a size of 1 / 8 may be extracted based on the basic structure feature subnetwork 201, the semantic features of the first image I1 with a size of 1 / 8 may be extracted based on the semantic feature subnetwork 202, and the edge features of the first image I1 with a size of 1 / 8 and the edge features of the first image I1 with a size of 1 / 4 (1 / 4H x 1 / 4W) may be extracted based on the edge feature subnetwork 203. In addition, in addition to the above image features, the feature extraction network 200 of the disparity estimation system 100 can also extract features based on the first image I1 itself in 1 / 2 size (1 / 2H x 1 / 2W), and features based on the first image I1 itself in full size (H x W), that is, the first image I1 itself.

[0071] The basic structural features of the first image I1 at a size of 1 / 8, the basic structural features of the second image I2 at a size of 1 / 8, the semantic features of the first image I1 at a size of 1 / 8, and the edge features of the first image I1 at a size of 1 / 8 can be output by the corresponding feature extraction subnetwork to the initial disparity generation subnetwork 301 for first-level disparity processing to obtain an initial disparity map dispS1 at a size of 1 / 8. Afterwards, the four disparity optimization subnetworks 302 sequentially cascaded with the initial disparity generation subnetwork 301 can sequentially perform different levels of disparity optimization processing on the initial disparity map dispS1 at a size of 1 / 8 based on the image features of corresponding sizes extracted by the feature extraction network 200, so as to obtain multiple optimized disparity maps of increasing sizes.

[0072] For example, the first disparity optimization subnetwork may perform disparity optimization processing on the 1 / 8-sized initial disparity map dispS1 output by the initial disparity generation subnetwork 301 according to the 1 / 8-sized edge features (partial or complete) of the first image I1 from the edge feature subnetwork 203, to obtain a 1 / 8-sized first optimized disparity map dispS1_refine. According to some embodiments, the first disparity optimization subnetwork may calculate a 1 / 8-sized first residual map based on the 1 / 8-sized edge features (partial or complete) of the first image I1 and the 1 / 8-sized initial disparity map dispS1, and add the 1 / 8-sized first residual map to the 1 / 8-sized initial disparity map dispS1 to obtain a 1 / 8-sized first optimized disparity map dispS1_refine.

[0073] The second disparity optimization subnetwork may perform disparity optimization processing on the 1 / 8-sized first optimized disparity map dispS1_refine output by the first disparity optimization subnetwork according to the 1 / 4-sized edge features (partial or complete) of the first image I1 from the edge feature subnetwork 203, to obtain a 1 / 4-sized second optimized disparity map dispS2_refine. According to some embodiments, the second disparity optimization subnetwork may upsample the 1 / 8-sized first optimized disparity map output by the first disparity optimization subnetwork to a 1 / 4-sized corresponding to the disparity processing at this level, and then calculate a 1 / 4-sized second residual map based on the 1 / 4-sized edge features (partial or complete) of the first image I1 and the upsampled 1 / 4-sized first optimized disparity map, and perform an addition operation on the 1 / 4-sized second residual map and the upsampled 1 / 4-sized first optimized disparity map to obtain a 1 / 4-sized second optimized disparity map dispS2_refine.

[0074] The third disparity optimization subnetwork may perform disparity optimization processing on the second optimized disparity map dispS2_refine of 1 / 4 size output by the second disparity optimization subnetwork according to the features (partial or complete) of 1 / 2 size based on the first image I1 itself extracted by the feature extraction network 200, to obtain a third optimized disparity map dispS3_refine of 1 / 2 size. According to some embodiments, the third disparity optimization subnetwork may upsample the second optimized disparity map of 1 / 4 size output by the second disparity optimization subnetwork to 1 / 2 size corresponding to the disparity processing at this level, and then calculate a third residual map of 1 / 2 size based on the features (partial or complete) of 1 / 2 size based on the first image itself and the upsampled second optimized disparity map of 1 / 2 size, and perform an addition operation on the third residual map of 1 / 2 size and the upsampled second optimized disparity map of 1 / 2 size to obtain a third optimized disparity map dispS3_refine of 1 / 2 size.

[0075] The fourth disparity optimization subnetwork may perform disparity optimization processing on the 1 / 2-sized third optimized disparity map dispS3_refine output by the third disparity optimization subnetwork according to the full-size features (partial or complete) based on the first image I1 itself extracted by the feature extraction network 200, to obtain a full-size fourth optimized disparity map dispS4_refine. According to some embodiments, the fourth disparity optimization subnetwork may upsample the 1 / 2-sized third optimized disparity map output by the third disparity optimization subnetwork to a full-size corresponding to the disparity processing at this level, and then calculate a full-size fourth residual map based on the full-size features (partial or complete) based on the first image itself and the upsampled full-size third optimized disparity map, and add the full-size fourth residual map and the upsampled full-size third optimized disparity map to obtain a full-size fourth optimized disparity map dispS4_refine. It should be noted that, in this example, the third and fourth disparity optimization sub-networks use features based on the first image itself to perform disparity optimization processing to reduce the amount of calculation, however, one or both of them can also use edge features or other features of the first image. Similarly, if it is necessary to further reduce the amount of calculation, the first and / or second disparity optimization sub-networks can also use features based on the first image itself instead of the extracted edge features, and so on.

[0076] The first optimized disparity map dispS1_refine of 1 / 8 size, the second optimized disparity map dispS2_refine of 1 / 4 size, the third optimized disparity map dispS3_refine of 1 / 2 size, and the fourth optimized disparity map dispS4_refine of full size can be used as Figure 5 The disparity estimation system 100 shown in FIG. 1 shows a plurality of disparity maps with successively increasing sizes.

[0077] in addition, Figure 6 The overall working process of the disparity estimation system 100 is similar to Figure 5 The overall working process of the disparity estimation system 100 shown is similar, the only difference is that the size of the initial disparity map generated by the initial disparity generation subnetwork 301 is smaller than the size of the optimized disparity map generated by the first disparity optimization subnetwork, and the initial disparity map generated by the initial disparity generation subnetwork 301 can be used as one of the multiple disparity maps with successively increasing sizes obtained by the disparity estimation system 100, so it will not be repeated here.

[0078] According to some embodiments, the initial disparity generation subnetwork 301 and each disparity optimization subnetwork 302 in the at least one disparity optimization subnetwork 302 may be any convolutional neural network that can implement the corresponding disparity processing function, such as 2DCNN (two-dimensional deep convolutional neural network) or 3DCNN (three-dimensional deep convolutional neural network). By using a convolutional neural network as a disparity processing subnetwork, a larger perception domain can be obtained, thereby improving the accuracy of the disparity map obtained by the disparity estimation system.

[0079] According to some embodiments, for the initial disparity generation subnetwork 301, if it adopts a 2DCNN structure to obtain disparity, the initial disparity generation subnetwork 301 may include a first number (e.g., 5, of course, the corresponding value may be flexibly selected according to actual needs) of sequentially cascaded convolution layers. The convolution method of each convolution layer may adopt, for example, depthwise separable convolution.

[0080] For example, the following table 1 will be used to describe a method that can be applied to Figure 5 The disparity estimation system shown in the figure is schematically illustrated by an initial disparity generation subnetwork 301 of a 2DCNN structure including five sequentially cascaded convolutional layers (e.g., conv1-conv5 in Table 1). As an example, the subnetwork adopts the MobileNetV2 network architecture.

[0081] Table 1: Description of a possible 2DCNN network structure of the initial disparity generation subnetwork 301

[0082]

[0083]

[0084] Combining Table 1 and Figure 5 It can be seen that the corr1d layer can be used to Figure 5 The feature extraction network 200 in the image extractor performs correlation operations on the 1 / 8 size basic structural features of the first image and the 1 / 8 size basic structural features of the second image. The semanS1_conv layer can be used to perform convolution processing on the 1 / 8 size semantic features of the first image extracted based on a 3x3 convolution kernel. The edgeS1_conv layer can be used to perform convolution processing on the 1 / 8 size edge features of the first image extracted based on a 3x3 convolution kernel. The concat layer can be used to merge the features output by corr1d, semanS1_conv and edgeS1_conv.

[0085] In addition, the MB_conv operation involved in the conv1-conv5 layer refers to the depthwise separable convolution operation in MobileNetV2, and the MB_conv_res operation refers to the residual depthwise separable convolution operation in MobileNetV2. In other words, the conv1 layer, conv2 layer, and conv4 layer can be used to perform depthwise separable convolution operations on the features output by the previous layer, and the conv3 layer and conv5 layer can be used to perform residual depthwise separable convolution operations on the features output by the previous layer. In addition, the dispS1 layer can be used to perform soft argmin calculations on the features output by the previous layer to obtain the initial disparity map dispS1 of the corresponding size (i.e., 1 / 8 size).

[0086] It should be noted that H and W mentioned in Table 1 may respectively represent the height and width of the image in the image pair input to the disparity estimation system 100, and D may represent the maximum disparity range of the image. The units of the three may be pixels, wherein the value of D may be related to the focal length of each camera in the multi-eye camera used to collect the image pair and / or the spacing between each camera. In addition, the number of convolutional layers of the initial disparity generation subnetwork 301 using the 2DCNN structure may be determined according to the number of features obtained by the concat layer. For example, if the number of features obtained by the concat layer is large, the number of convolutional layers included in the initial disparity generation subnetwork 301 may also be increased.

[0087] As an alternative, the initial disparity generation subnetwork 301 may also adopt a 3DCNN structure to obtain disparity. The initial disparity generation subnetwork 301 adopting the 3DCNN structure may include a second number (for example, 7, and of course the corresponding value may be flexibly selected according to actual needs) of convolutional layers cascaded in sequence.

[0088] For example, the following table 2 will be used to describe a method that can be applied to Figure 5 The disparity estimation system shown is schematically illustrated with an initial disparity generation subnetwork 301 of a 3DCNN structure including seven sequentially cascaded convolutional layers (eg, conv1-conv7 in Table 2).

[0089] Table 2: Description of a possible 3DCNN network structure of the initial disparity generation subnetwork 301

[0090]

[0091]

[0092] Combining Table 2 and Figure 5 It can be seen that the edgeS1_conv layer can be used to perform convolution processing on the edge features of the 1 / 8 size of the extracted first image based on the 3x3 convolution kernel. The semanS3_conv layer can be used to perform convolution processing on the semantic features of the 1 / 8 size of the extracted first image based on the 3x3 convolution kernel. The concat layer can be used to merge the features output by featS1, semanS1_conv and edgeS1_conv, where, although not shown in Table 2, featS1 can refer to the basic structural features of the 1 / 8 size of the extracted first image and the basic structural features of the 1 / 8 size of the second image.

[0093] In addition, the cost layer can be used to translate the features output by the concat layer. The conv1 to conv7 layers can be used to perform convolution operations on the features output by the previous layer based on a 3x3x3 convolution kernel. The conv2, conv4, and conv6 layers are equivalent to the residual modules of the 3DCNN network, and can also be used to perform convolution operations on the features output by the previous layer and then add the convolution results to the results output by the previous layer. The dispS1 layer can be used to perform soft argmin calculations on the features output by the previous layer to obtain an initial disparity map dispS1 of the corresponding size (i.e., 1 / 8 size).

[0094] Similar to Table 1, H and W mentioned in Table 2 may respectively represent the height and width of an image in an image pair input to the disparity estimation system 100. In addition, F may represent the number of channels of a feature, 1F represents the number of channels as F, 3F represents the number of channels as 3x F, and so on. In addition, the number of convolutional layers of the initial disparity generation subnetwork 301 using the 3DCNN structure may also be determined according to the number of features obtained by the concat layer. For example, if the number of features obtained by the concat layer is large, the number of convolutional layers included in the initial disparity generation subnetwork 301 may also be increased.

[0095] According to some embodiments, the number of convolutional layers included in each disparity optimization subnetwork 302 in the at least one disparity optimization subnetwork 302 may be less than the number of convolutional layers included in the initial disparity generation subnetwork 301. For example, taking each disparity optimization subnetwork 302 adopting a 2DCNN structure as an example, the number of convolutional layers included in each disparity optimization subnetwork 302 may be 3, and of course, it may also be set to other values ​​according to actual needs. In addition, referring to the description of the aforementioned embodiment, each disparity optimization subnetwork 302 may also adopt a 3DCNN structure, which is not limited thereto.

[0096] The following table 3 to table 6 will be used to list the applicable Figure 5The structure of multiple disparity optimization sub-networks 302 of the disparity estimation system shown in FIG. 3 is schematically illustrated, wherein Tables 3 to 6 respectively describe Figure 5 Possible 2DCNN network structures of the first to fourth disparity optimization subnetworks of the disparity estimation system shown are described accordingly.

[0097] Table 3: Description of a possible 2DCNN network structure for the first disparity optimization subnetwork

[0098]

[0099] Similar to the above Table 1, the edgeS1_conv layer can be used to perform convolution processing on the 1 / 8 size edge features of the extracted first image based on the 3x3 convolution kernel. The concat layer can be used to merge the 1 / 8 size initial disparity map dispS1 generated by the previous level of disparity processing (ie, the initial disparity generation processing) and the features output by edgeS1_conv.

[0100] In addition, the conv1 layer and the conv3 layer can be used to perform depth-separable convolution operations on the features output by the previous layer, and the conv2 layer can be used to perform residual depth-separable convolution operations on the features output by the previous layer. In addition, the dispS1_refine layer can be used to perform superposition operations on the features output by the previous conv3 layer and the 1 / 8 size initial disparity map dispS1 generated by the previous level of disparity processing to obtain the first optimized disparity map dispS1_refine of the corresponding size (i.e., 1 / 8 size).

[0101] Table 4: Description of a possible 2DCNN network structure for the second disparity optimization subnetwork

[0102]

[0103]

[0104] From Table 4 and Figure 5 It can be seen that the dispS1_up layer can be used to upsample the 1 / 8 size first optimized disparity map dispS1_refine generated by the previous level of disparity processing (i.e., the first level of disparity optimization processing) to obtain a 1 / 4 size optimized disparity map dispS1_up. The edgeS2_conv layer can be used to perform convolution processing on the 1 / 4 size edge features of the extracted first image based on a 3x3 convolution kernel. The concat layer can be used to merge the 1 / 4 size optimized disparity map dispS1_up after upsampling and the features output by edgeS2_conv.

[0105] In addition, the conv1 layer and the conv3 layer can be used to perform depth-separable convolution operations on the features output by the previous layer, and the conv2 layer can be used to perform residual depth-separable convolution operations on the features output by the previous layer. In addition, the dispS2_refine layer can be used to perform superposition operations on the features output by the previous conv3 layer and the 1 / 4 size optimized disparity map dispS1_up after upsampling to obtain a second optimized disparity map dispS2_refine of the corresponding size (i.e., 1 / 4 size).

[0106] Table 5: Description of a possible 2DCNN network structure for the third disparity optimization subnetwork

[0107]

[0108] From Table 5 and Figure 5 It can be seen that the dispS2_up layer can be used to upsample the 1 / 4 size second optimized disparity map dispS2_refine generated by the previous level of disparity processing (i.e., the second level of disparity optimization processing) to obtain a 1 / 2 size optimized disparity map dispS2_up. The imgS3 layer can be used to downsample the first image itself to obtain a 1 / 2 size feature based on the first image itself, where I1 in Table 5 represents the first image. The concat layer can be used to merge the 1 / 2 size optimized disparity map dispS2_up after upsampling and the features output by imgS3.

[0109] In addition, the conv1 layer, the conv2 layer, and the conv3 layer can be used to perform convolution operations on the features output by the previous layer, and the dispS3_refine layer can be used to perform superposition operations on the features output by the previous conv3 layer and the 1 / 2-size optimized disparity map dispS2_up after upsampling to obtain a third optimized disparity map dispS3_refine of the corresponding size (i.e., 1 / 2 size).

[0110] Table 6: Description of a possible 2DCNN network structure for the fourth disparity optimization subnetwork

[0111]

[0112]

[0113] From Table 6 and Figure 5It can be seen that the dispS3_up layer can be used to upsample the 1 / 2 size third optimized disparity map dispS3_refine generated by the previous level of disparity processing (i.e., the third level of disparity optimization processing) to obtain a full-size optimized disparity map dispS3_up. The concat layer can be used to merge the full-size optimized disparity map dispS3_up after the upsampling process and the first image itself, where I1 in Table 6 represents the first image.

[0114] In addition, the conv1 layer, the conv2 layer, and the conv3 layer can be used to perform convolution operations on the features output by the previous layer, and the dispS4_refine layer can be used to perform superposition operations on the features output by the previous conv3 layer and the full-size optimized disparity map dispS3_up after upsampling to obtain a fourth optimized disparity map dispS4_refine of the corresponding size (ie, full size).

[0115] It should be noted that, similar to the above-mentioned embodiments, H and W mentioned in Tables 3 to 6 may respectively represent the height and width of the image in the image pair input to the disparity estimation system 100. In addition, the number of convolutional layers of each disparity optimization sub-network 302 using the 2DCNN structure may also be determined according to the number of features obtained by the concat layer. For example, if the number of features obtained by the concat layer is large, the number of convolutional layers included in each disparity optimization sub-network 302 may also be increased.

[0116] According to some embodiments, each sub-network in the initial disparity generation sub-network 301 and the at least one disparity optimization sub-network 302 may be a network pre-trained based on a training sample set, so that the efficiency of disparity processing can be improved. Of course, according to actual needs, each sub-network in the initial disparity generation sub-network 301 and the at least one disparity optimization sub-network 302 may also be obtained by real-time training based on a training sample set, or may also be obtained by real-time or timed optimization of a pre-trained network based on an updated training sample set, so as to improve the accuracy of disparity generation.

[0117] According to some embodiments, the training process of each sub-network in the initial disparity generation sub-network 301 and the at least one disparity optimization sub-network 302 may also adopt supervised training or unsupervised training, which can be flexibly selected according to actual needs. The introduction of supervised training and unsupervised training can refer to the relevant description in the aforementioned relevant embodiments, which will not be repeated here.

[0118] According to some embodiments, each of the initial disparity generation subnetwork 301 and the at least one disparity optimization subnetwork 302 may also be configured to calculate a loss function, which may be used to represent the error between the disparity in the disparity map generated by the subnetwork and the corresponding true disparity. In this way, by calculating the loss function, the accuracy of each disparity map generated by the disparity estimation system may be clarified. In addition, the corresponding system may also be optimized based on the loss function.

[0119] According to some embodiments, taking the initial disparity generation subnetwork 301 and each of the at least one disparity optimization subnetwork 302 using supervised training as an example, the loss function of each disparity processing subnetwork or each level of disparity processing output can be defined as L n =f(Disp GTn -Disp Sn )+g(Disp Sn ). The value of n ranges from 1 to N (corresponding to Figure 5 The disparity estimation system shown), or 0 to N (corresponding to Figure 6 The function f represents the predicted disparity (Disp Sn ) and the true disparity (Disp GTn ), and g represents the parallax continuity constraint. g(x)=|x x |+|x y |. In addition, edge features may also be considered as regularization terms of the loss function, which is not limited. Accordingly, the final loss function of the disparity estimation system 100 may be the sum of the loss functions of the disparity processing subnetworks or disparity processing outputs at all levels.

[0120] According to some other embodiments, if the initial disparity generation subnetwork 301 and each subnetwork in the at least one disparity optimization subnetwork 302 adopt unsupervised training, the loss function of each disparity processing subnetwork or each level of disparity processing can be obtained by reconstructing the image and calculating the reconstruction error. For example, taking the calculation of the loss function of one level of disparity processing subnetwork as an example, the calculated loss function can be expressed as Wherein, warpI1=warp(I2,Disp1), the warp function indicates that the second image I2 is reconstructed into an I1 image according to the disparity calculated by the disparity processing subnetwork of this level.

[0121] In the following, the initial disparity generation subnetwork 301 and each subnetwork in the at least one disparity optimization subnetwork 302 are trained using supervised training, the training set uses Scene Flow, and the structure of the disparity estimation system is as follows: Figure 5 As an example, combined Fig. 7A , Figure 7B as well as Figure 8 The reference image on which the training is based, the corresponding true disparity map, and the results obtained by applying the trained parameters to the Middlebury dataset images are schematically illustrated.

[0122] Fig. 7A and Figure 7B Schematic diagrams showing reference images and corresponding ground truth disparity maps based on which network training is based according to an exemplary embodiment of the present disclosure, respectively, Figure 8 The disparity estimation system after training is shown according to an exemplary embodiment of the present disclosure. Fig. 7A Schematic diagram of multiple disparity maps with increasing sizes from right to left obtained by cascading multi-level disparity processing of the reference image shown in FIG. 1 (i.e., the result obtained by applying the trained parameters to the Middlebury dataset image). As can be seen from the above figures, the sizes of the obtained multiple disparity maps can be increased successively, the accuracy can be increased successively, and the accuracy of the disparity map with the largest size is close to the true value disparity map. In addition, although Fig. 7A , Figure 7B as well as Figure 8 The reference image, the true disparity map and the generated multiple disparity maps are illustrated in grayscale images. It can be understood that when Fig. 7A When the reference image shown is a color image, Figure 7B as well as Figure 8 Each disparity map shown may also be a corresponding color image.

[0123] According to some embodiments, the disparity generation network 300 may also be configured to select, from the multiple disparity maps, a disparity map whose size matches the performance of the target device as the disparity map provided to the target device, according to the performance of the target device. For example, if the performance of the target device is high, and / or the accuracy of the disparity map required by it is high, a disparity map with a larger size may be selected from the multiple disparity maps and provided to the target device. In addition, the target device may also actively obtain the disparity map required by it from the multiple disparity maps obtained by the disparity estimation system according to its own performance, and there is no limitation on this.

[0124] In addition, although not shown, the multiple disparity maps obtained by the disparity estimation system can also be provided to the corresponding target device for further processing, for example, provided to the corresponding target device so that the target device can calculate the depth map based on the disparity map, and then obtain the depth information of the scene, so as to be applied to various application scenarios such as three-dimensional reconstruction, autonomous driving, obstacle detection, etc.

[0125] The above has been combined Figures 1 to 8An exemplary system for disparity estimation according to the present disclosure is described. Fig. 9 , Fig.10 as well as Fig.11 The exemplary method of disparity estimation and the exemplary electronic device of the present disclosure are further described. Figures 1 to 8 The various definitions, embodiments, implementations, examples, etc. described may also be applied to or combined with the exemplary embodiments described later.

[0126] According to some embodiments, Fig. 9 FIG. 4 is a flowchart of a parallax estimation method according to an exemplary embodiment of the present disclosure. Fig. 9 As shown, the disparity estimation method of the present disclosure may include the following steps: extracting features from each image in an image pair (step S901); and performing cascaded multi-level disparity processing based on the extracted image features to obtain multiple disparity maps of successively increasing sizes (step S902), wherein the input of the first level of disparity processing in the multi-level disparity processing includes multiple image features with sizes corresponding to the disparity processing of this level; the input of each level of disparity processing in the multi-level disparity processing except the first level of disparity processing includes: one or more image features with sizes corresponding to the disparity processing of this level, and the disparity map generated by the previous level of disparity processing.

[0127] According to some embodiments, the image pair may be an image pair for the same scene captured by a multi-camera. The sizes of the images in the image pair are the same, but the corresponding viewing angles are different. In addition, the images in the image pair may be grayscale images or color images.

[0128] According to some embodiments, the extracted image features of each image in the image pair may include at least one or more of the following features: basic structural features, semantic features, edge features, texture features, color features, object shape features, or features based on the image itself. For example, the extracted image features of the first image in the image pair (e.g., the left image) may include basic structural features, semantic features, and edge features, and the extracted image features of the second image in the image pair (e.g., the right image) may include basic structural features. Alternatively, the extracted image features of the first image and the second image in the image pair may both include basic structural features, semantic features, and edge features, and so on.

[0129] According to some embodiments, the disparity map with the largest size among the multiple disparity maps may be consistent with the size of each image in the image pair. Of course, the size of each disparity map in the multiple disparity maps may also be smaller than the size of each image in the image pair. In addition, the height and width of the latter disparity map in any two adjacent disparity maps among the multiple disparity maps may be twice the height and width of the previous disparity map, respectively. Of course, they may also be set to 3 times, 4 times, or other positive integer multiples greater than 1, respectively, of the height and width of the previous disparity map according to the actual required accuracy. For example, taking the case where there are four disparity maps and the size of the last disparity map in the plurality of disparity maps is H x W (which may be consistent with the size of each image in the image pair), the sizes of the other disparity maps in the plurality of disparity maps before it may be 1 / 2H x 1 / 2W (if H x W may be called full size, 1 / 2H x 1 / 2W may be called 1 / 2 size), 1 / 4H x 1 / 4W (may be called 1 / 4 size), and 1 / 8H x 1 / 8W (may be called 1 / 8 size).

[0130] According to some embodiments, the extracted image features may include image features of N sizes, where N is a positive integer not less than 2. Fig.10 As shown ( Fig.10 A flowchart of multi-level disparity processing in a disparity estimation method according to an exemplary embodiment of the present disclosure is shown, and cascaded multi-level disparity processing is performed according to the extracted image features to obtain multiple disparity maps with successively increasing sizes, which may include the following steps.

[0131] Step S1001: in the first level of disparity processing of the multi-level disparity processing, generating an initial disparity map having the minimum size according to at least a portion of image features of the minimum size among the image features of the N sizes.

[0132] For example, taking the extracted N sizes of image features including image features of four sizes: 1 / 8 size (1 / 8H x 1 / 8W), 1 / 4 size (1 / 4H x 1 / 4W), 1 / 2 size (1 / 2H x 1 / 2W) and full size (H x W) as an example, in the first-level disparity processing of the multi-level disparity processing, an initial disparity map with the smallest size (i.e., 1 / 8 size) can be generated based on at least a portion of the image features of the smallest size (i.e., 1 / 8 size) among the four sizes of image features.

[0133] In addition, as shown in Tables 1 and 2 above, the corresponding image features with corresponding sizes can be parallax-translated and superimposed, and 3DCNN can be used to obtain an initial disparity map, or the difference of the image features with corresponding sizes after translation can be calculated, and 2DCNN can be used to obtain an initial disparity map.

[0134] Step S1002: In each subsequent level of disparity processing of the multi-level disparity processing, disparity optimization processing is performed on the disparity map generated by the previous level of disparity processing according to at least a portion of the image features with corresponding sizes among the image features of the N sizes, so as to generate an optimized disparity map with the corresponding size, wherein the multiple disparity maps at least include each optimized disparity map.

[0135] According to some embodiments, the multi-level disparity processing may include N+1 level disparity processing. Accordingly, in each subsequent level of disparity processing of the multi-level disparity processing, according to at least a portion of the image features with corresponding sizes among the image features of the N sizes, a disparity optimization process is performed on the disparity map generated by the previous level of disparity processing to generate an optimized disparity map with the corresponding size, wherein the multiple disparity maps at least include each optimized disparity map, which may include:

[0136] In the N levels of disparity processing other than the first level of disparity processing, disparity optimization processing is performed on the disparity map generated by the previous level of disparity processing in order of size from small to large, based on at least a part of the image features with corresponding sizes among the image features of the N sizes, to obtain N optimized disparity maps with successively increasing sizes, and the N optimized disparity maps are used as the multiple disparity maps, wherein the sizes of the N optimized disparity maps respectively correspond to the N sizes.

[0137] For example, the extracted N sizes of image features include image features of four sizes, namely, 1 / 8 size (1 / 8H x 1 / 8W), 1 / 4 size (1 / 4H x 1 / 4W), 1 / 2 size (1 / 2H x 1 / 2W) and full size (H x W), and the multi-level disparity processing may include 4+1 level disparity processing. In the four levels of disparity processing except the first level disparity processing, disparity optimization processing may be performed on the disparity map generated by the previous level of disparity processing in order of size from small to large, based on at least a part of the image features of the four sizes with corresponding sizes, to obtain four optimized disparity maps with successively increasing sizes (for example, an optimized disparity map with 1 / 8 size, an optimized disparity map with 1 / 4 size, an optimized disparity map with 1 / 2 size and an optimized disparity map with full size), and the four optimized disparity maps are used as the multiple disparity maps.

[0138] According to some other embodiments, the multi-level disparity processing may include N-level disparity processing. Accordingly, in each subsequent level of disparity processing of the multi-level disparity processing, according to at least a portion of image features with corresponding sizes among the image features of the N sizes, a disparity optimization process is performed on the disparity map generated by the previous level of disparity processing to generate an optimized disparity map with the corresponding size, wherein the multiple disparity maps at least include each optimized disparity map, which may include:

[0139] In the N-1 levels of disparity processing other than the first level of disparity processing, in order of size from small to large, disparity optimization processing is performed on the disparity map generated by the previous level of disparity processing based on at least a part of the image features with corresponding sizes among the N-1 non-minimum sizes of the image features of the N sizes, to obtain N-1 optimized disparity maps with successively increasing sizes, and the initial disparity map and the N-1 optimized disparity maps are used as the multiple disparity maps, wherein the sizes of the initial disparity map and the N-1 optimized disparity maps correspond to the N sizes, respectively.

[0140] For example, taking the case where the extracted N sizes of image features include image features of four sizes, namely, 1 / 8 size (1 / 8H x 1 / 8W), 1 / 4 size (1 / 4H x 1 / 4W), 1 / 2 size (1 / 2H x 1 / 2W) and full size (H x W), and the multi-level disparity processing may include four levels of disparity processing, in the other three levels of disparity processing except the first level of disparity processing, disparity optimization processing may be performed on the disparity map generated by the previous level of disparity processing in order of size from small to large, based on at least a part of the image features of corresponding sizes in the other three non-minimum sized image features, to obtain three optimized disparity maps of increasing size in sequence (for example, an optimized disparity map with 1 / 4 size, an optimized disparity map with 1 / 2 size and an optimized disparity map with full size), and the initial disparity map and the three optimized disparity maps are used as the multiple disparity maps.

[0141] Thus, the obtained multiple disparity maps may include or exclude the initial disparity map generated by the first level disparity processing in the multi-level disparity processing, so as to improve the flexibility of disparity generation.

[0142] According to some embodiments, in each subsequent level of disparity processing of the multi-level disparity processing, performing disparity optimization processing on the disparity map generated by the previous level of disparity processing according to at least a portion of the image features with corresponding sizes among the image features of the N sizes to generate an optimized disparity map with the corresponding size may include:

[0143] In each level of disparity processing except the first level of disparity processing in the multi-level disparity processing, residual calculation processing is performed on the disparity map generated by the previous level of disparity processing based on at least a part of the image features with a corresponding size to obtain a residual map with the corresponding size, and the residual map with the corresponding size and the disparity map generated by the previous level of disparity processing are combined to obtain an optimized disparity map with the corresponding size.

[0144] For example, taking the case where the extracted image features of N sizes include image features of four sizes, namely, 1 / 8 size (1 / 8H x 1 / 8W), 1 / 4 size (1 / 4H x 1 / 4W), 1 / 2 size (1 / 2H x 1 / 2W) and full size (H x W), and the multi-level disparity processing may include 4+1 level disparity processing, in the disparity processing corresponding to the 1 / 8 size of the other four levels of disparity processing except the first level of disparity processing (i.e., the disparity optimization processing corresponding to the 1 / 8 size), a first residual map of 1 / 8 size can be calculated based on part or all of the extracted image features of 1 / 8 size and the initial disparity map generated by the previous level of disparity processing, and a first optimized disparity map of 1 / 8 size can be calculated based on the first residual map and the initial disparity map. In the next level of disparity optimization processing corresponding to 1 / 4 size, a second residual map of 1 / 4 size can be calculated based on part or all of the extracted 1 / 4 size image features and the first optimized disparity map generated by the previous level of disparity processing, and a second optimized disparity map of 1 / 4 size can be calculated based on the second residual map and the first optimized disparity map, and so on.

[0145] According to some embodiments, in each level of disparity processing except the first level of disparity processing in the multi-level disparity processing, before performing disparity optimization processing on the disparity map generated by the previous level of disparity processing, the method may further include: in response to the size of the disparity map generated by the previous level of disparity processing being smaller than the size corresponding to the current level of disparity processing, upsampling the disparity map generated by the previous level of disparity processing to a size corresponding to the current level of disparity processing.

[0146] For example, still taking the extracted N sizes of image features including 4 sizes of 1 / 8 size (1 / 8H x 1 / 8W), 1 / 4 size (1 / 4H x 1 / 4W), 1 / 2 size (1 / 2H x 1 / 2W) and full size (H x W), and the multi-level disparity processing may include 4+1 level disparity processing as an example, in the disparity processing corresponding to the 1 / 4 size of the other 4 levels of disparity processing except the first level of disparity processing (i.e., the disparity optimization processing corresponding to the 1 / 4 size), the first optimized disparity map of 1 / 8 size generated by the previous level of disparity processing may be upsampled to the 1 / 4 size corresponding to the current level of disparity processing, and then, based on part or all of the extracted 1 / 4 size image features, the upsampled first optimized disparity map of 1 / 4 size may be subjected to disparity optimization processing to obtain a second optimized disparity map of 1 / 4 size.

[0147] According to some embodiments, the image feature of the smallest size among the N sizes of image features may, for example, include at least one image feature of the first image in the image pair and at least one image feature of the second image. For example, the image feature of the smallest size among the N sizes of image features may include basic structural features, semantic features, and edge features of the first image (e.g., the left image) in the image pair and basic structural features of the second image (e.g., the right image) in the image pair.

[0148] Each non-minimum-sized image feature among the N sizes of image features may, for example, include at least one image feature of the first image in the image pair and / or at least one image feature of the second image. For example, each non-minimum-sized image feature among the N sizes of image features may include an edge feature of the first image in the image pair or a feature based on the first image itself.

[0149] In addition, referring to the aforementioned system embodiments, the image features based on which different optimized disparity maps are generated may be image features of the same type or image features of different types; and / or, the image features based on which different optimized disparity maps are generated may be image features of the same image or different images in the image pair.

[0150] The image features based on which each optimized disparity map is generated may include, for example, edge features of at least one image in the image pair, and / or features of the image itself based on at least one image in the image pair. The features of the image itself based on at least one image in the image pair may include, for example, the image itself of the at least one image, or an image obtained by downsampling the image itself of the at least one image according to the size of the optimized disparity map to be generated.

[0151] According to some embodiments, the disparity estimation method may further include: calculating a loss function of each level of disparity processing in the multi-level disparity processing, wherein the loss function may be used to represent an error between the disparity in the disparity map generated by the level of disparity processing and the corresponding true disparity. Thus, by calculating the loss function, the accuracy of each disparity map generated may be clarified, and the disparity estimation method may be optimized based on the loss function.

[0152] According to some embodiments, the disparity estimation method may further include: selecting, according to the performance of the target device, from the multiple disparity maps a disparity map whose size matches the performance of the target device as the disparity map provided to the target device. For example, if the performance of the target device is high, and / or the accuracy of the disparity map required by it is high, a disparity map with a larger size may be selected from the multiple disparity maps and provided to the target device. In addition, the target device may also actively obtain the disparity map it needs from the multiple disparity maps obtained by the disparity estimation system according to its own performance.

[0153] In addition, the disparity estimation method may further include: before performing image feature extraction on each image in the image pair, performing epipolar correction on the images in the image pair so that the images in the image pair have disparity only in one direction (e.g., horizontal direction). Thus, the disparity search range of the image can be limited to one direction, thereby improving the efficiency of subsequent feature extraction and disparity generation.

[0154] One aspect of the present disclosure may include an electronic device, which may include a processor; and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform any of the aforementioned methods.

[0155] One aspect of the present disclosure may include a computer-readable storage medium storing a program including instructions that, when executed by a processor of an electronic device, cause the electronic device to perform any of the aforementioned methods.

[0156] Reference Fig.11 , a computing device 2000 will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The computing device 2000 can be any machine configured to perform processing and / or computing, and can be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal digital assistant, a smart phone, a vehicle-mounted computer, or any combination thereof. The above electronic devices can be implemented in whole or in part by the computing device 2000 or a similar device or system.

[0157] The computing device 2000 may include an element connected to or in communication with the bus 2002 (possibly via one or more interfaces). For example, the computing device 2000 may include a bus 2002, one or more processors 2004, one or more input devices 2006, and one or more output devices 2008. The one or more processors 2004 may be any type of processor, and may include, but are not limited to, one or more general-purpose processors and / or one or more special-purpose processors (e.g., special processing chips). The input device 2006 may be any type of device capable of inputting information to the computing device 2000, and may include, but are not limited to, a mouse, a keyboard, a touch screen, a microphone, and / or a remote controller. The output device 2008 may be any type of device capable of presenting information, and may include, but are not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The computing device 2000 may also include or be connected to a non-transitory storage device 2010, which may be any storage device that is non-transitory and can implement data storage, and may include but is not limited to a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a tape or any other magnetic medium, an optical disk or any other optical medium, a ROM (read-only memory), a RAM (random access memory), a cache memory and / or any other memory chip or cartridge, and / or any other medium from which a computer can read data, instructions and / or code. The non-transitory storage device 2010 may be detachable from the interface. The non-transitory storage device 2010 may have data / programs (including instructions) / code for implementing the above methods and steps. The computing device 2000 may also include a communication device 2012. The communication device 2012 can be any type of device or system that enables communication with an external device and / or with a network, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication device and / or a chipset, such as a Bluetooth™ device, a 1302.11 device, a WiFi device, a WiMax device, a cellular communication device and / or the like.

[0158] The computing device 2000 may also include a working memory 2014, which may be any type of working memory that can store programs (including instructions) and / or data useful for the operation of the processor 2004, and may include, but is not limited to, random access memory and / or read-only memory devices.

[0159] The software elements (programs) may be located in the working memory 2014, including but not limited to an operating system 2016, one or more applications 2018, a driver and / or other data and code. Instructions for executing the above methods and steps may be included in one or more applications 2018, and the feature extraction network 200 and the disparity generation network 300 of the above disparity estimation system 100 may be implemented by the processor 2004 reading and executing the instructions of one or more applications 2018. More specifically, the feature extraction network 200 of the above disparity estimation system 100 may be implemented, for example, by the processor 2004 executing the application 2018 having instructions for executing step S901. In addition, the disparity generation network 300 of the above disparity estimation system 100 may be implemented, for example, by the processor 2004 executing the application 2018 having instructions for executing step S902, and so on. The executable code or source code of the instructions of the software element (program) can be stored in a non-transitory computer-readable storage medium (such as the storage device 2010 described above), and when executed, it can be stored in the working memory 2014 (possibly compiled and / or installed). The executable code or source code of the instructions of the software element (program) can also be downloaded from a remote location.

[0160] It should also be understood that various modifications may be made according to specific requirements. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. For example, some or all of the disclosed methods and apparatus may be implemented by programming hardware (e.g., programmable logic circuits including field programmable gate arrays (FPGAs) and / or programmable logic arrays (PLAs)) in assembly language or hardware programming languages ​​(such as VERILOG, VHDL, C++) using logic and algorithms according to the present disclosure.

[0161] It should also be understood that the foregoing method can be implemented through a server-client model. For example, the client can receive data input by the user and send the data to the server. The client can also receive data input by the user, perform a portion of the processing in the foregoing method, and send the processed data to the server. The server can receive data from the client, and execute the foregoing method or another portion of the foregoing method, and return the execution result to the client. The client can receive the execution result of the method from the server, and present it to the user, for example, through an output device.

[0162] It should also be understood that the components of computing device 2000 can be distributed across a network. For example, some processing can be performed using one processor while other processing can be performed by another processor remote from the one processor. Other components of computing system 2000 can also be similarly distributed. In this way, computing device 2000 can be interpreted as a distributed computing system that performs processing at multiple locations.

[0163] Some exemplary aspects of the disclosure are described below.

[0164] Aspect 1. A disparity estimation system, comprising:

[0165] a feature extraction network configured to extract features from each image in the image pair and output the extracted image features to the disparity generation network; and

[0166] The disparity generation network is configured to perform cascaded multi-level disparity processing according to the extracted image features to obtain a plurality of disparity maps with successively increasing sizes,

[0167] Among them, the input of the first level of disparity processing in the multi-level disparity processing includes multiple image features with sizes corresponding to the disparity processing of this level; the input of each level of disparity processing except the first level of disparity processing in the multi-level disparity processing includes: one or more image features with sizes corresponding to the disparity processing of this level, and the disparity map generated by the previous level of disparity processing.

[0168] Aspect 2. The disparity estimation system according to aspect 1, wherein the extracted image features of each image in the image pair include at least one or more of the following features:

[0169] Basic structural features, semantic features, edge features, texture features, color features, object shape features, or features based on the image itself.

[0170] Aspect 3. A disparity estimation system as described in Aspect 2, wherein the feature extraction network includes multiple feature extraction sub-networks respectively used to extract different features of the image; the multiple feature extraction sub-networks include at least a basic structure feature sub-network, a semantic feature sub-network and an edge feature sub-network.

[0171] Aspect 4. The disparity estimation system according to aspect 1, wherein the disparity map having the largest size among the plurality of disparity maps is consistent with the size of each image in the image pair.

[0172] Aspect 5. The disparity estimation system according to aspect 1, wherein:

[0173] The extracted image features include image features of N sizes, where N is a positive integer not less than 2;

[0174] The disparity generation network is configured to generate an initial disparity map having the minimum size according to at least a portion of image features of the minimum size among the image features of the N sizes in the first level disparity processing of the multi-level disparity processing; and

[0175] In each subsequent level of disparity processing of the multi-level disparity processing, disparity optimization processing is performed on the disparity map generated by the previous level of disparity processing according to at least a portion of the image features with corresponding sizes among the image features of the N sizes, so as to generate an optimized disparity map with the corresponding size;

[0176] The multiple disparity maps at least include various optimized disparity maps.

[0177] Aspect 6. The disparity estimation system according to aspect 5, wherein:

[0178] The multi-level parallax processing includes N+1 level parallax processing;

[0179] The disparity generation network is configured to, in N levels of disparity processing other than the first level of disparity processing, perform disparity optimization processing on the disparity map generated by the previous level of disparity processing in order of size from small to large, based on at least a portion of the image features with corresponding sizes among the image features of the N sizes, to obtain N optimized disparity maps with successively increasing sizes, and use the N optimized disparity maps as the multiple disparity maps, wherein the sizes of the N optimized disparity maps correspond to the N sizes, respectively.

[0180] Aspect 7. The disparity estimation system according to aspect 5, wherein:

[0181] The multi-level parallax processing includes N-level parallax processing;

[0182] The disparity generation network is configured to, in N-1 levels of disparity processing other than the first level of disparity processing, perform disparity optimization processing on the disparity map generated by the previous level of disparity processing in order of size from small to large, based on at least a portion of image features with corresponding sizes among N-1 types of non-minimum sizes among the image features of the N types of sizes, to obtain N-1 optimized disparity maps with successively increasing sizes, and use the initial disparity map and the N-1 optimized disparity maps as the multiple disparity maps, wherein the sizes of the initial disparity map and the N-1 optimized disparity maps correspond to the N sizes, respectively.

[0183] Aspect 8. A disparity estimation system as described in any one of aspects 5 to 7, wherein:

[0184] The disparity generation network is configured to, in each level of disparity processing except the first level of disparity processing in the multi-level disparity processing, perform residual calculation processing on the disparity map generated by the previous level of disparity processing based on at least a portion of the image features with a corresponding size to obtain a residual map with the corresponding size, and combine the residual map with the corresponding size and the disparity map generated by the previous level of disparity processing to obtain an optimized disparity map with the corresponding size.

[0185] Aspect 9. The disparity estimation system as described in any one of Aspects 5-7, wherein:

[0186] The disparity generation network is configured to, in each level of disparity processing except the first level of disparity processing in the multi-level disparity processing, before performing disparity optimization processing on the disparity map generated by the previous level of disparity processing, upsample the disparity map generated by the previous level of disparity processing to a size corresponding to the current level of disparity processing in response to the size of the disparity map generated by the previous level of disparity processing being smaller than the size corresponding to the current level of disparity processing.

[0187] Aspect 10. A disparity estimation system as described in any one of Aspects 5-7, wherein the image feature of the smallest size among the N sizes of image features includes at least one image feature of the first image in the image pair and at least one image feature of the second image, and each non-minimum-sized image feature among the N sizes of image features includes at least one image feature of the first image in the image pair and / or at least one image feature of the second image.

[0188] Aspect 11. A disparity estimation system as described in any one of Aspects 5-7, wherein the image features based on which different optimized disparity maps are generated are image features of the same type or image features of different types; and / or,

[0189] The image features based on which different optimized disparity maps are generated are image features of the same image or different images in the image pair.

[0190] Aspect 12. A disparity estimation system as described in any one of Aspects 5-7, wherein the image features based on which each optimized disparity map is generated include edge features of at least one image in the image pair, and / or features of the image itself based on at least one image in the image pair.

[0191] Aspect 13. A disparity estimation system as described in Aspect 12, wherein the characteristics of the image itself based on at least one image in the image pair include the image itself of the at least one image, or an image obtained by downsampling the image itself of the at least one image according to the size of the optimized disparity map to be generated.

[0192] Aspect 14. A disparity estimation system as described in Aspect 1, wherein the disparity generation network includes an initial disparity generation subnetwork and at least one disparity optimization subnetwork, the initial disparity generation subnetwork and each disparity optimization subnetwork in the at least one disparity optimization subnetwork are cascaded in sequence, the initial disparity generation subnetwork is configured to perform the first-level disparity processing, and the at least one disparity optimization subnetwork is configured to perform various levels of disparity processing except the first-level disparity processing.

[0193] Aspect 15. The disparity estimation system according to aspect 14, wherein:

[0194] Each of the initial disparity generation subnetwork and the at least one disparity optimization subnetwork is a two-dimensional deep convolutional neural network 2DCNN or a three-dimensional deep convolutional neural network 3DCNN.

[0195] Aspect 16. The disparity estimation system according to Aspect 15, wherein the number of convolutional layers included in each of the at least one disparity optimization subnetworks is smaller than the number of convolutional layers included in the initial disparity generation subnetwork.

[0196] Aspect 17. A disparity estimation system as described in Aspect 14, wherein the initial disparity generation subnetwork and each of the at least one disparity optimization subnetwork are configured to calculate a loss function, which represents the error between the disparity in the disparity map generated by the subnetwork and the corresponding true disparity.

[0197] Aspect 18. The disparity estimation system according to aspect 1, wherein:

[0198] The disparity generation network is configured to select, according to the performance of the target device, from the multiple disparity maps a disparity map whose size matches the performance of the target device as the disparity map provided to the target device.

[0199] Aspect 19. The disparity estimation system as described in Aspect 1, wherein the image pair is an image pair of the same scene captured by multiple cameras.

[0200] Aspect 20. A disparity estimation method, comprising:

[0201] performing feature extraction on each image in the image pair; and

[0202] According to the extracted image features, cascaded multi-level disparity processing is performed to obtain multiple disparity maps with successively increasing sizes.

[0203] Among them, the input of the first level of disparity processing in the multi-level disparity processing includes multiple image features with sizes corresponding to the disparity processing of this level; the input of each level of disparity processing except the first level of disparity processing in the multi-level disparity processing includes: one or more image features with sizes corresponding to the disparity processing of this level, and the disparity map generated by the previous level of disparity processing.

[0204] Aspect 21. The disparity estimation method according to Aspect 20, wherein the extracted image features of each image in the image pair include at least one or more of the following features:

[0205] Basic structural features, semantic features, edge features, texture features, color features, object shape features, or features based on the image itself.

[0206] Aspect 22. The disparity estimation method according to Aspect 20, wherein the disparity map having the largest size among the plurality of disparity maps is consistent with the size of each image in the image pair.

[0207] Aspect 23. The disparity estimation method according to aspect 20, wherein:

[0208] The extracted image features include image features of N sizes, where N is a positive integer not less than 2;

[0209] The method of performing cascaded multi-level disparity processing according to the extracted image features to obtain a plurality of disparity maps with successively increasing sizes includes:

[0210] In the first level of the multi-level disparity processing, generating an initial disparity map having the minimum size according to at least a portion of image features of the minimum size among the image features of the N sizes; and

[0211] In each subsequent level of disparity processing of the multi-level disparity processing, disparity optimization processing is performed on the disparity map generated by the previous level of disparity processing according to at least a portion of the image features with corresponding sizes among the image features of the N sizes, so as to generate an optimized disparity map with the corresponding size;

[0212] The multiple disparity maps at least include various optimized disparity maps.

[0213] Aspect 24. The disparity estimation method according to aspect 23, wherein:

[0214] The multi-level parallax processing includes N+1 level parallax processing;

[0215] In each subsequent level of disparity processing of the multi-level disparity processing, disparity optimization processing is performed on the disparity map generated by the previous level of disparity processing according to at least a portion of the image features with corresponding sizes among the image features of the N sizes, so as to generate an optimized disparity map with the corresponding size, wherein the multiple disparity maps at least include each optimized disparity map, including:

[0216] In the N levels of disparity processing other than the first level of disparity processing, disparity optimization processing is performed on the disparity map generated by the previous level of disparity processing in order of size from small to large, based on at least a part of the image features with corresponding sizes among the image features of the N sizes, to obtain N optimized disparity maps with successively increasing sizes, and the N optimized disparity maps are used as the multiple disparity maps, wherein the sizes of the N optimized disparity maps respectively correspond to the N sizes.

[0217] Aspect 25. The disparity estimation method according to aspect 23, wherein:

[0218] The multi-level parallax processing includes N-level parallax processing;

[0219] In each subsequent level of disparity processing of the multi-level disparity processing, disparity optimization processing is performed on the disparity map generated by the previous level of disparity processing according to at least a portion of the image features with corresponding sizes among the image features of the N sizes, so as to generate an optimized disparity map with the corresponding size, wherein the multiple disparity maps at least include each optimized disparity map, including:

[0220] In the N-1 levels of disparity processing other than the first level of disparity processing, in order of size from small to large, disparity optimization processing is performed on the disparity map generated by the previous level of disparity processing based on at least a part of the image features with corresponding sizes among the N-1 non-minimum sizes of the image features of the N sizes, to obtain N-1 optimized disparity maps with successively increasing sizes, and the initial disparity map and the N-1 optimized disparity maps are used as the multiple disparity maps, wherein the sizes of the initial disparity map and the N-1 optimized disparity maps correspond to the N sizes, respectively.

[0221] Aspect 26. The disparity estimation method as described in any one of Aspects 23-25, wherein:

[0222] In each subsequent level of disparity processing of the multi-level disparity processing, disparity optimization processing is performed on the disparity map generated by the previous level of disparity processing according to at least a portion of the image features with corresponding sizes among the image features of the N sizes, so as to generate an optimized disparity map with the corresponding size, including:

[0223] In each level of disparity processing except the first level of disparity processing in the multi-level disparity processing, residual calculation processing is performed on the disparity map generated by the previous level of disparity processing based on at least a part of the image features with a corresponding size to obtain a residual map with the corresponding size, and the residual map with the corresponding size and the disparity map generated by the previous level of disparity processing are combined to obtain an optimized disparity map with the corresponding size.

[0224] Aspect 27. The disparity estimation method according to any one of aspects 23 to 25, wherein, in each level of disparity processing except the first level of disparity processing in the multi-level disparity processing, before performing disparity optimization processing on the disparity map generated by the previous level of disparity processing, the method further comprises:

[0225] In response to the size of the disparity map generated by the previous level of disparity processing being smaller than the size corresponding to the current level of disparity processing, the disparity map generated by the previous level of disparity processing is upsampled to a size corresponding to the current level of disparity processing.

[0226] Aspect 28. A disparity estimation method as described in any one of Aspects 23-25, wherein the image feature of the smallest size among the N sizes of image features includes at least one image feature of the first image in the image pair and at least one image feature of the second image, and each non-minimum-sized image feature among the N sizes of image features includes at least one image feature of the first image in the image pair and / or at least one image feature of the second image.

[0227] Aspect 29. The disparity estimation method according to any one of Aspects 23 to 25, wherein the image features based on which different optimized disparity maps are generated are image features of the same type or image features of different types; and / or,

[0228] The image features based on which different optimized disparity maps are generated are image features of the same image or different images in the image pair.

[0229] Aspect 30. A disparity estimation method as described in any one of Aspects 23-25, wherein the image features based on which each optimized disparity map is generated include edge features of at least one image in the image pair, and / or features of the image itself based on at least one image in the image pair.

[0230] Aspect 31. A disparity estimation method as described in Aspect 30, wherein the characteristics of the image itself based on at least one image in the image pair include the image itself of the at least one image, or an image obtained by downsampling the image itself of the at least one image according to the size of the optimized disparity map to be generated.

[0231] Aspect 32. The disparity estimation method according to aspect 20, further comprising:

[0232] A loss function of each level of disparity processing in the multi-level disparity processing is calculated, where the loss function represents an error between the disparity in the disparity map generated by the level of disparity processing and the corresponding real disparity.

[0233] Aspect 33. The disparity estimation method according to aspect 20, further comprising:

[0234] According to the performance of the target device, a disparity map whose size matches the performance of the target device is selected from the multiple disparity maps as the disparity map provided to the target device.

[0235] Aspect 34. The disparity estimation method as described in Aspect 20, wherein the image pair is an image pair of the same scene acquired by multiple cameras.

[0236] Aspect 35. An electronic device comprising:

[0237] Processor; and

[0238] A memory storing a program, the program comprising instructions which, when executed by the processor, cause the processor to perform a method according to any one of aspects 20-34.

[0239] Aspect 36. A computer-readable storage medium storing a program, the program comprising instructions, which, when executed by a processor of an electronic device, cause the electronic device to perform the method according to any one of aspects 20-34.

[0240] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A disparity estimation system, comprising: A feature extraction network is configured to extract features from each image in the image pair and output the extracted image features to the disparity generation network; as well as The disparity generation network is configured to perform cascaded multi-level disparity processing according to the extracted image features to obtain a plurality of disparity maps with successively increasing sizes, The input of the first level of disparity processing in the multi-level disparity processing includes a plurality of image features having a size corresponding to the disparity processing at the level; the input of each level of disparity processing except the first level of disparity processing in the multi-level disparity processing includes: one or more image features having a size corresponding to the disparity processing at the level, and a disparity map generated by the previous level of disparity processing; The extracted image features include image features of N sizes, where N is a positive integer not less than 2, and the multi-level parallax processing includes N+1 level parallax processing; and The disparity generation network is configured to, in the first level of disparity processing of the multi-level disparity processing, generate an initial disparity map with the minimum size based on at least a part of the image features of the minimum size among the image features of the N sizes, and in the N levels of disparity processing other than the first level of disparity processing, in order of size from small to large, sequentially based on at least a part of the image features with corresponding sizes among the image features of the N sizes including the image features of the minimum size, perform disparity optimization processing on the disparity map generated by the previous level of disparity processing to obtain N optimized disparity maps with successively increasing sizes, and use the obtained N optimized disparity maps as the disparity maps with successively increasing sizes, wherein the sizes of the N optimized disparity maps respectively correspond to the N sizes including the minimum size.

2. The disparity estimation system of claim 1, wherein: The extracted image features of each image in the image pair include at least one or more of the following features: Basic structural features, semantic features, edge features, texture features, color features, object shape features, or features based on the image itself.

3. The disparity estimation system of claim 2, wherein: The feature extraction network includes multiple feature extraction subnetworks respectively used to extract different features of the image; the multiple feature extraction subnetworks include at least a basic structure feature subnetwork, a semantic feature subnetwork and an edge feature subnetwork.

4. The disparity estimation system of claim 1, wherein: A disparity map having a largest size among the plurality of disparity maps is consistent with a size of each image in the image pair.

5. The disparity estimation system of claim 1, wherein: The disparity generation network is configured to, in each level of disparity processing except the first level of disparity processing in the multi-level disparity processing, perform residual calculation processing on the disparity map generated by the previous level of disparity processing based on at least a portion of the image features with a corresponding size to obtain a residual map with the corresponding size, and combine the residual map with the corresponding size and the disparity map generated by the previous level of disparity processing to obtain an optimized disparity map with the corresponding size.

6. The disparity estimation system of claim 1, wherein: The disparity generation network is configured to, in each level of disparity processing except the first level of disparity processing in the multi-level disparity processing, before performing disparity optimization processing on the disparity map generated by the previous level of disparity processing, upsample the disparity map generated by the previous level of disparity processing to a size corresponding to the current level of disparity processing in response to the size of the disparity map generated by the previous level of disparity processing being smaller than the size corresponding to the current level of disparity processing.

7. The disparity estimation system of claim 1, wherein: The image feature of the smallest size among the N sizes of image features includes at least one image feature of the first image in the image pair and at least one image feature of the second image, and each non-minimum size of the image features of the N sizes includes at least one image feature of the first image and / or at least one image feature of the second image in the image pair.

8. The disparity estimation system of claim 1, wherein: The image features based on which different optimized disparity maps are generated are the same type of image features or different types of image features; and / or, The image features based on which different optimized disparity maps are generated are image features of the same image or different images in the image pair.

9. The disparity estimation system of claim 1, wherein: The image features based on which each optimized disparity map is generated include edge features of at least one image in the image pair and / or features of the image itself based on at least one image in the image pair.

10. The disparity estimation system of claim 9, wherein: The feature based on the image itself of at least one image in the image pair includes the image itself of the at least one image, or an image obtained by downsampling the image itself of the at least one image according to a size of the optimized disparity map to be generated.

11. The disparity estimation system of claim 1, wherein: The disparity generation network includes an initial disparity generation subnetwork and at least one disparity optimization subnetwork. The initial disparity generation subnetwork and each disparity optimization subnetwork in the at least one disparity optimization subnetwork are cascaded in sequence. The initial disparity generation subnetwork is configured to perform the first-level disparity processing, and the at least one disparity optimization subnetwork is configured to perform various levels of disparity processing except the first-level disparity processing.

12. The disparity estimation system of claim 11, wherein: Each of the initial disparity generation subnetwork and the at least one disparity optimization subnetwork is a two-dimensional deep convolutional neural network 2DCNN or a three-dimensional deep convolutional neural network 3DCNN.

13. The disparity estimation system of claim 12, wherein: The number of convolutional layers included in each of the at least one disparity optimization sub-networks is smaller than the number of convolutional layers included in the initial disparity generation sub-network.

14. The disparity estimation system of claim 11, wherein: The initial disparity generation subnetwork and each of the at least one disparity optimization subnetwork are configured to calculate a loss function, wherein the loss function represents an error between a disparity in a disparity map generated by the subnetwork and a corresponding true disparity.

15. The disparity estimation system of claim 1, wherein: The disparity generation network is configured to select, according to the performance of the target device, from the multiple disparity maps a disparity map whose size matches the performance of the target device as the disparity map provided to the target device.

16. The disparity estimation system of claim 1, wherein: The image pair is an image pair of the same scene captured by a multi-camera.

17. A disparity estimation method, comprising: Performing feature extraction on each image in the image pair; as well as According to the extracted image features, cascaded multi-level disparity processing is performed to obtain multiple disparity maps with successively increasing sizes. The input of the first level of disparity processing in the multi-level disparity processing includes a plurality of image features having a size corresponding to the disparity processing at the level; the input of each level of disparity processing except the first level of disparity processing in the multi-level disparity processing includes: one or more image features having a size corresponding to the disparity processing at the level, and a disparity map generated by the previous level of disparity processing; The extracted image features include image features of N sizes, where N is a positive integer not less than 2, and the multi-level parallax processing includes N+1 level parallax processing; and The method of performing cascaded multi-level disparity processing according to the extracted image features to obtain a plurality of disparity maps with successively increasing sizes includes: In the first-level disparity processing of the multi-level disparity processing, an initial disparity map with the minimum size is generated based on at least a part of the image features of the minimum size among the image features of the N sizes, and in the N levels of disparity processing other than the first-level disparity processing, in order of size from small to large, disparity optimization processing is performed on the disparity map generated by the previous level of disparity processing based on at least a part of the image features with corresponding sizes among the image features of the N sizes including the image features of the minimum size, to obtain N optimized disparity maps with successively increasing sizes, and the obtained N optimized disparity maps are used as the disparity maps with successively increasing sizes, wherein the sizes of the N optimized disparity maps respectively correspond to the N sizes including the minimum size.

18. The disparity estimation method according to claim 17, wherein: The extracted image features of each image in the image pair include at least one or more of the following features: Basic structural features, semantic features, edge features, texture features, color features, object shape features, or features based on the image itself.

19. The disparity estimation method according to claim 17, wherein: A disparity map having a largest size among the plurality of disparity maps is consistent with a size of each image in the image pair.

20. The disparity estimation method according to claim 17, wherein: In each subsequent level of disparity processing of the multi-level disparity processing, disparity optimization processing is performed on the disparity map generated by the previous level of disparity processing according to at least a portion of the image features with corresponding sizes among the image features of the N sizes, so as to generate an optimized disparity map with the corresponding size, including: In each level of disparity processing except the first level of disparity processing in the multi-level disparity processing, residual calculation processing is performed on the disparity map generated by the previous level of disparity processing based on at least a part of the image features with a corresponding size to obtain a residual map with the corresponding size, and the residual map with the corresponding size and the disparity map generated by the previous level of disparity processing are combined to obtain an optimized disparity map with the corresponding size.

21. The disparity estimation method according to claim 17, wherein: In each level of disparity processing except the first level of disparity processing in the multi-level disparity processing, before performing disparity optimization processing on the disparity map generated by the previous level of disparity processing, the method further includes: In response to the size of the disparity map generated by the previous level of disparity processing being smaller than the size corresponding to the current level of disparity processing, the disparity map generated by the previous level of disparity processing is upsampled to a size corresponding to the current level of disparity processing.

22. The disparity estimation method according to claim 17, wherein: The image feature of the smallest size among the N sizes of image features includes at least one image feature of the first image in the image pair and at least one image feature of the second image, and each non-minimum size of the image features of the N sizes includes at least one image feature of the first image and / or at least one image feature of the second image in the image pair.

23. The disparity estimation method according to claim 17, wherein: The image features based on which different optimized disparity maps are generated are the same type of image features or different types of image features; and / or, The image features based on which different optimized disparity maps are generated are image features of the same image or different images in the image pair.

24. The disparity estimation method according to claim 17, wherein: The image features based on which each optimized disparity map is generated include edge features of at least one image in the image pair and / or features of the image itself based on at least one image in the image pair.

25. The disparity estimation method according to claim 24, wherein: The feature based on the image itself of at least one image in the image pair includes the image itself of the at least one image, or an image obtained by downsampling the image itself of the at least one image according to a size of the optimized disparity map to be generated.

26. The disparity estimation method of claim 17, further comprising: A loss function of each level of disparity processing in the multi-level disparity processing is calculated, where the loss function represents an error between the disparity in the disparity map generated by the level of disparity processing and the corresponding real disparity.

27. The disparity estimation method of claim 17, further comprising: According to the performance of the target device, a disparity map whose size matches the performance of the target device is selected from the multiple disparity maps as the disparity map provided to the target device.

28. The disparity estimation method according to claim 17, wherein: The image pair is an image pair of the same scene captured by a multi-camera.

29. An electronic device comprising: processor; as well as A memory storing a program, the program comprising instructions which, when executed by the processor, cause the processor to perform a method according to any one of claims 17-28.

30. A computer-readable storage medium storing a program, the program comprising instructions, which, when executed by a processor of an electronic device, cause the electronic device to perform the method according to any one of claims 17-28.

31. A computer program product comprising program code portions for performing the method according to any one of claims 17 to 28 when the computer program product is run on one or more computing devices.

Images