ARTIFICIAL INTELLIGENCE STEREO DISPARITY ESTIMATION

Abstract

Systems and methods for a computer-based stereo disparity estimation are disclosed. The methods include generating a cost-volume matrix based on a stereo image pair. The methods include generating a disparity map for the first image of the stereo image pair, which, for each pixel in the first image, generates a disparity value corresponding to the pixel by performing stereo image processing on the cost-volume matrix entry corresponding to the pixel to generate an intermediate output of the stereo image processing, generating one or more weight values ​​using the intermediate output of the stereo image processing as input for a computational neural network (CNN), and calculating the disparity value for the pixel using one or more intermediate disparity values ​​from the intermediate output of the stereo image processing and the multiple weight values.

Description

TECHNICAL AREA

[0001] At least one embodiment relates to image processing and in particular the use of artificial intelligence (AI) to generate a disparity map for a stereo image pair. BACKGROUND

[0002] Dense stereo matching, or stereo mapping, is an image processing technique that estimates the depth of each pixel in a pair of images taken from slightly different positions. This is achieved by identifying points in the two images that correspond to the same position, a phenomenon known as disparity. The disparity value represents the positional difference between the corresponding pixels in the two images. These disparity values ​​can be arranged in a disparity map, a two-dimensional image where the value of each pixel corresponds to the disparity value at a corresponding pixel in the original images. Dense stereo matching (and the resulting disparity map) allows for the estimation of distances to various points using only a single pair of images, thus eliminating the need for remote sensing technologies such as lidar, sonar, or radar. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a block diagram of an exemplary system for stereo disparity estimation using artificial intelligence (Kl) according to at least one embodiment. Fig. Figure 2 is a block diagram of an example of a stereo image assigner for a Kl stereo diparity estimation according to at least one embodiment. Fig. Figure 3 shows an example of census transformation and Hamming distance calculations performed from a cost / volume constructor. Fig. 2 can be carried out according to some exemplary embodiments. Fig. Figure 4 is a block diagram of an exemplary SGM / eSGM block for a Kl stereodiparity estimation according to at least one embodiment. Fig. 5A illustrates exemplary path directions that are configurable for SGM / eSGM calculations and can be used in some exemplary embodiments. Fig. Figure 5B shows several iterations of SGM / eSGM implementations according to some embodiments. Fig. Figure 6 is a block diagram of an exemplary Kl model for a Kl stereodiparity estimation according to at least one embodiment. Fig. Figure 7 is an exemplary data flow diagram for a Kl stereodiparity estimation according to at least one embodiment. DETAILED DESCRIPTION

[0003] There are several methods for dense stereo matching, such as semi-global matching (SGM) and efficient semi-global matching (eSGM). To calculate a disparity value for a pixel in an image, SGM and eSGM generate multiple paths to that pixel, each path containing disparity values, from which one is selected as the disparity value for the pixel. A drawback of SGM and eSGM is that they assign equal weight or importance to each path, which is usually incorrect. For example, if the image has a texture in the horizontal direction, the paths along the horizontal direction should be weighted more heavily than the other paths. For this reason, SGM and eSGM produce lower-quality disparity estimates for pixels corresponding to small or thin vertical areas, such as posts or signposts, or for pixels corresponding to areas where the texture has strong alignment, such as...B. a road surface.

[0004] Aspects of the present disclosure address the aforementioned and other shortcomings by providing a stereo image mapper with an artificial intelligence (AI) model that uses intermediate outputs from SGM or eSGM to determine disparity values ​​for pixels, rather than using disparity values ​​estimated by SGM or eSGM. The AI ​​model can be a convolutional neural network (CNN) trained to output a weight associated with each path to a pixel generated by SGM or eSGM. Disparity values ​​from the paths can be combined using the associated weights to determine the disparity value for the pixel. The determined disparity values ​​can be used to generate a disparity map.

[0005] Advantages of the disclosed embodiments over the existing technology include, but are not limited to, increased accuracy for disparity maps for use in determining distances to different points, especially for areas where conventional dense stereo matching has yielded poor results.

[0006] Fig. Figure 1 schematically represents a system for Kl stereo diparity estimation according to some exemplary embodiments. The depicted System 100 can be a computing device, a system-on-a-chip (SoC), or another type of device that incorporates a specialized stereo image matching circuit in the form of a stereo image matcher. The System 100 can be used in various implementations. For example, the System 100 can be part of a vehicle system (including an autonomous or semi-autonomous vehicle) capable of object / pedestrian detection / tracking, structure-from-motion (SFM) determination, simultaneous localization and mapping (SLAM), etc. The System 100 can be used with virtual reality applications, for example, for stitching 360-degree videos. The System 100 can be used in gaming applications, for example, for frame rate upconversion.The System 100 can be used with deep learning applications, such as video classification. The System 100 can also be used in other applications that utilize stereo disparity.

[0007] In some embodiments, the system 100 includes a stereo image mapper 102. The stereo image mapper 102 may include one or more processors, processing units, or other circuits configured at least to generate a stereo disparity map from input images and associated input information. As above and below with regard to the Fig. As mentioned in 2-7, the Stereo Image Mapping 102 implements an SGM or an eSGM to generate intermediate outputs, and the Stereo Image Mapping 102 implements a trained CNN that uses the intermediate outputs to generate stereo disparity determinations (e.g., in the form of a disparity map corresponding to the input images).

[0008] A graphics processing unit (GPU) 106 can be connected directly and / or indirectly to the stereo image assigner 102 via a graphics host 104. The graphics host 104 provides a programming and control interface to various graphics and video engines and to a display interface(s). The graphics host 104 can also provide interfaces (in Fig. (1 not shown) to a switch (e.g., a crossbar switch or the like) for connecting to other components and a direct memory interface for retrieving commands and / or command structures from system memory. In some embodiments, commands and / or command structures are either collected from a push buffer in memory or provided directly by the central processing unit (CPU) 108 and then delivered to clients also connected to the graphics host, such as the stereo image mapper 102. An audio / video image encoder / decoder 112 is connected via the graphics host 104. The audio / video image encoder / decoder 112 can support the playback and / or generation of high-resolution full-screen video (e.g., 1440p high definition) in any format, such as H.264 BP / MP / HP / MMC, VC-1, VP8, MPEG-2, or MPEG-4.

[0009] The stereo image assigner 102 can receive its input images from a memory and its output images into memory (in Fig. (1 not shown) write, for example, a frame buffer memory accessed via a frame buffer interface 110. Many components in the system 100, including, for example, the GPU 106, the stereo image mapper 102, the video encoder / decoder 112, or the display interface 114, can be connected to the frame buffer interface 110 to access the frame buffer.

[0010] The CPU 108 can control the processing on the system 100 and can be connected to the GPU 106. The CPU 108 and the GPU 106 can be connected to a memory controller 116 to access external memory.

[0011] In an exemplary embodiment, if the system 100 is integrated, for example, in an automotive application, incoming video material from one or more cameras mounted on the automobile (or another vehicle) can be received by the video encoder / decoder 112, which decodes the video material and writes the video frames to the frame buffer (not shown) via the frame buffer interface 110. The video frames are then retrieved from the frame buffer by the stereo image mapper 102 to generate a disparity map, which is supplied to the GPU 106 via the frame buffer. The GPU 106 can use the generated disparity map for further processing in any application, for example, but not limited to, object detection and / or tracking.

[0012] In some embodiments, the GPU 106 or the CPU 108 can perform one or more of the operations described herein that are performed by the stereo image assigner 102. For example, the GPU 106 or the CPU 108 can receive executable instructions for the one or more operations of the stereo image assigner 102 from the memory controller 116, and the GPU 106 or the CPU 108 can execute these instructions.

[0013] Fig. Figure 2 schematically shows the exemplary circuit of the stereo image assigner 102, which is in Fig. 1 is shown, according to some embodiments. Fig. Figure 2 shows the stereo image mapper 102 connected to the graphics host 104. The circuit of the stereo image mapper 102 can include a microcontroller 202, an image buffer interface 204 (which extends from the image buffer interface 110 in Fig. 1 can be distinguished from or be identical to it), an SGM / eSGM block 206, a CVC block (Cost Volume Constructor) 208, an RPC block (Reference Pixel Cache) 210, an RPF block (Reference Pixel Fetch) 212, and a CPF block (Current Pixel Fetch) 214, as well as an AI block 216. The AI ​​block 216 can include a KL model 218 and a disparity value calculator 220.

[0014] The microcontroller 202 can be connected to the graphics host 104, from which it receives commands and data. The microcontroller 202 can connect multiple components in the stereo image folder 102 to control the operations in the stereo image folder 102 according to the commands received from the graphics host 104.

[0015] The microcontroller 202 can have interfaces for signals such as context switch signals, microcode for specific instructions, addresses and other data, a privilege bus, and an interrupt interface with the graphics host 104. It can process the microcode, addresses, data, and / or other signals it receives and control the rest of the stereo image assigner 102. The microcontroller 202 can also handle error and perform other tasks, such as rate control and general (e.g., macroblock-level) management tasks, tracking, and mode decision configuration. The microcontroller 202 can receive interrupt requests, status data, or control data from the AI ​​block 216.

[0016] The image buffer interface 204 enables the stereo image assigner 102 to output from an image buffer (e.g., the image buffer interface 110 in Fig. 1) to read and write to it. For example, data such as the images input into the stereo image folder 102 can be read from the stereo image folder 102 via the image buffer interface 204 according to control signals received from the microcontroller 202. The disparity maps generated as output from the stereo image folder 102 can be written to the image buffer via the image buffer interface 204.

[0017] The SGM / eSGM block 206 can include circuits for one-dimensional (1D) and / or two-dimensional (2D) SGM / eSGM operations, the generation of historical and / or temporary path costs, and a winner decision. The SGM / eSGM block 206 can also support aspects of post-processing. The SGM / eSGM block 206 can be configured to perform 1D or 2D SGM / eSGM along a configurable number of paths (e.g., 4 or 8 paths). The SGM / eSGM processing can also be configured for different disparity levels (e.g., 128 or 256 disparities) for stereo SGM / eSGM and epipolar SGM / eSGM. The "Disparity Levels" parameter can define the search space used for matching. This means that if the disparity level is D, for each pixel p in the base image, D pixels in the reference image are sought for matching, creating D disparity levels that are linked to p.

[0018] The SGM / eSGM block 206 can, in some embodiments, implement one, none, or all of the following: equiangular subpixel interpolation, adaptive smoothing penalties, and wavefront processing (e.g., for bandwidth saving). Equiangular subpixel interpolation can be performed for subpixel refinement and, in some embodiments, can be enabled or disabled based on a configuration parameter. The SGM / eSGM block 206 can provide a unified architecture for stereo disparity and configurable scalability between quality and performance. The SGM / eSGM block 206 can also provide configurable motion vector / disparity granularity (e.g., minimum 1x1 to maximum 8x8), a configurable number of disparity levels and search areas, and / or cost calculation on the original resolution to achieve matching accuracy.Further details regarding the SGM / eSGM block 206 are provided below in relation to the . Fig. 3, 4 and 5A-C provided.

[0019] As part of performing SGM / eSGM operations, the SGM / eSGM block 206 can generate an intermediate stereo image processing output. The SGM / eSGM block 206 can supply the intermediate stereo image processing output to the Kl block 216, as described below with respect to Fig. Section 6 is explained.

[0020] The CVC block 208 can include a circuit that can be operated to generate the cost volume according to the input images. The "cost volume" (also called "matching cost volume" or "assignment cost volume") is a three-dimensional (3D) arrangement in which each element represents the matching cost of a pixel at a specific disparity level. The in Fig. Cost volume matrix 318 shown is an example. The CVC block 208 can be configured to perform a variety of operations, including performing a census transformation (e.g., a 5x5 census transformation) on current and reference pixels and calculating the Hamming distance between census-transformed data blocks for current and reference pixels, as shown below with respect to Fig. 3 explained.

[0021] CPF block 214 may contain circuitry that can be operated to obtain a current pixel or the next pixel to be evaluated. RPC block 210 and RPF block 212 may contain circuitry that can be operated to obtain and store the reference pixels corresponding to each pixel fetched by CPF block 214. RPC block 210 may have a cache for storing reference pixels and may reduce the memory bandwidth required for retrieving reference pixels. RPC block 210 may accept the retrieval request from RPF block 212, retrieve the reference pixels from external memory, and output the reference pixel block to CVC block 208.

[0022] The AI ​​block 216 can receive the intermediate output of the stereo image processing from the SGM / eSGM block 206. The AI ​​block 216 can supply the intermediate output of the stereo image processing as input to the AI ​​model 218, and the AI ​​model 218 can generate one or more weight values. The disparity value calculator 220 can use the one or more weight values ​​and parts of the intermediate output of the stereo image processing (e.g., intermediate disparity values) to calculate a disparity value for a given pixel. The disparity value calculator 220 can generate the disparity map using the calculated disparity values ​​for the pixels and supply the disparity map to the image buffer interface 204. Alternatively, the disparity value calculator 220 can supply the calculated disparity values ​​to the image buffer interface 204, and a separate component can generate the disparity map using the calculated disparity values.Information on how the Kl-Block 216 calculates the disparity values ​​is given below in relation to . Fig. 6 given.

[0023] As an exemplary overview of the AI ​​stereo disparity estimation procedure implemented by SGM / eSGM block 206, CVC block 208, and Kl block 216 (and supported by the various other components of the stereo image assigner 102), CVC block 208 generates a cost volume matrix containing costs corresponding to each pixel in the first image of an input stereo image pair. CVC block 208 provides the cost volume matrix to SGM / eSGM block 206. For each pixel in the first image of the stereo image pair, SGM / eSGM block 206 performs stereo image processing using the entry in the cost volume matrix corresponding to that pixel to generate an intermediate output of the stereo image processing for that pixel.The intermediate output from the stereo image processing is then passed to the KL block 216, which uses the KL model 218 and the disparity value calculator 220 to calculate a disparity value for the current pixel (instead of the disparity value generated by the SGM / eSGM block 206). The stereo image assigner 102 can repeat the stereo image processing and the KL operations for each pixel in the first image to generate a disparity value for each pixel in the first image. The disparity values ​​are then organized in a disparity map, which the System 100 can use in various applications.

[0024] Fig. Figure 3 shows an example of census transformation and Hamming distance calculations performed from CVC block 208. Fig. 2 according to some exemplary embodiments. The pixel block 302, which in this example is a 5x5 pixel block, can be the current pixel block that is retrieved when the CPF block 214 retrieves the center pixel x as the current pixel. The value of each pixel in the retrieved pixel block can represent an intensity value.

[0025] The census transformation, which can be used in some embodiments, is a nonlinear transformation that maps a local neighborhood around a pixel P, specified as pixel block 302, to a binary string 306 representing the set of neighboring pixels whose intensity is less than that of P, specified as pixel block 304. Each census digit ξ(P, P') is defined by the following relationship: ξ(P,P')={0,P>P'1,P≤P'}

[0026] This means that for a pixel P, each pixel P' in its neighborhood is represented as 1 or 0, depending on whether P' is greater than, equal to, or less than P. The size of the local neighborhood of pixel P for the census transform can be configured. Based on a trade-off between output quality and chip area, some exemplary embodiments use a 5x5 census transform in CVC block 208. The binary string 306 is derived from the census-transformed block 304 by linearly ordering the rows from top to bottom.

[0027] For each pixel P, the binary string 306, representing the set of adjacent pixels for two images, is then subjected to a Hamming distance determination, as shown in 316. The Hamming distance is a distance metric used to measure the difference between two-bit string values. In the context of CVC block 208, the Hamming distance is the number of distinct bits in two census transform strings. The Hamming distance for pixel P can be determined by XORing the two-bit strings and then counting the number of 1s.

[0028] The census transformation result arrays 310 and 312 represent census transformation results for the corresponding left and right stereo images, respectively, according to an example. The census transformation result array 310 can be viewed as the collection of census-transformed results (i.e., the binary strings 306 corresponding to the respective pixel of the image) for each pixel in the left image. Similarly, the census transformation result array 312 can be viewed as the collection of census transformation results for the respective pixel in the right image. Figure 314 shows an example of the current pixel p with its bit sequence in the left image and a reference pixel with its bit sequence in the right image.

[0029] Figure 316 shows the calculation of the Hamming distance by performing an XOR operation on the census-transformed results taken from the left and right images, as explained above. Arrays 310 and 312, containing the census-transformed results, are compared according to the following equation: C(x,y,d)=∑x',y'∈[x±2,y±2]Hamming Distance(CT0(x',y'),CT1(x'−d,y')) to generate a 3D disparity space, referred to as cost volume matrix 318. CT0 is the result array 310 of the census transformation and CT1 is the result array 312 of the census transformation. The CVC 208 delivers the cost volume matrix 318 to the SGM / eSGM block 206 for use in stereo image processing.

[0030] Fig. Figure 4 is a schematic block diagram of an exemplary SGM / eSGM block 206 according to some embodiments. In the stereo image assigner 102, the SGM / eSGM block 206 can be the subunit that receives the cost volume matrix 318 from the CVC block 208, performs SGM / eSGM operations, and post-processes the resulting disparity values ​​(e.g., the winner disparity value). SGM / eSGM are dynamically programmed algorithms used for stereo disparity estimation.

[0031] The cost volume matrix 318 from the CVC block 208 is received by a path cost calculator 402. The path cost calculator 402 can be configured to use the cost volume matrix 318 to calculate a path cost value along a path to a pixel of the input stereo image pair. The path cost calculator 402 can use at least one section of the cost volume matrix 318 to calculate one or more path cost values ​​for the current pixel. The path cost calculator 402 can use a previously calculated path cost value to calculate a current path cost value. The path cost calculator 402 can receive the previously calculated path cost values ​​from the path cost buffer 404, which can store one or more previously calculated path cost values. The calculation of the one or more path cost values ​​is described below with respect to Fig. 5A explained.

[0032] The one or more path cost values ​​calculated by the path cost calculator 402 can be provided to the path cost buffer 404 for storage. The one or more path cost values ​​can also be provided to the winner determination block 406. The winner determination block 406 can select the path cost value containing the winning disparity and output one or more path cost values ​​and / or one or more disparity values, as shown below with respect to the Fig. 5B is explained.

[0033] The output of the winner determination block 406 can be provided to the post-processing block 408. Post-processing block 408 can perform post-processing operations, which may include error correction, subpixel interpolation, VZ index-to-motion vector conversion, disparity-to-motion vector conversion, or other post-processing operations. After post-processing by post-processing block 408, the result can be returned to the winner determination block 406. The result can then be passed to the KL block 216.

[0034] Features supported by the SGM / eSGM block 206 in some embodiments include support for a configurable maximum number of possible disparity values ​​(e.g., 256 or 128 disparities, with the lower number of disparities being selectable for faster processing speed). Other supported features may include a configurable number of directions in which matching costs are evaluated, for example, 2 (horizontal and vertical), 4 (horizontal, vertical, left, and right), or 8 (horizontal, vertical, left, right, and the four diagonals), as well as support for a configurable number of SGM passes (e.g., 1, 2, or 3).

[0035] Fig. Figure 5A illustrates exemplary path directions for SGM / eSGM that can be used in some embodiments. In some embodiments, the number of paths 504 considered when determining the path cost for a pixel ρ 502 can be configurable. For example, in the illustrated figure 506, the matching cost value associated with pixel p 502 can be determined based on four paths (e.g., top L2, bottom L6, left L0, and right L4) or eight paths (e.g., L0-L7). In some embodiments, SGM / eSGM can use a different subset of the eight paths L0-L7 and / or additional paths.

[0036] In some embodiments, the path cost calculator 402 can calculate the path cost value L for pixel p along a direction r for d disparity levels as follows: Lr(p,d)=C(p,d)+temp(p,d)−miniLr(p−r,i) temp(p,d)=min{Lr(p−r,i)minLr(p−r,d±1)+P1miniLr(p−r,i)+P2

[0037] In the recursive calculation above, the path cost L for a pixel p along a path r is determined using all path costs from the previous pixel along direction r (represented as "pr") and two penalty terms P1 and P2. C(p, d) is the sum of all pixel-matching costs for the disparities of d. temp(p, d) adds a constant penalty P1 for all pixels in the neighborhood of p for which the disparity changes by a small amount (e.g., 1 pixel). i L r(pr, i) adds a larger constant penalty P2 for all larger disparity changes. Using a smaller penalty for small changes allows for adaptation to sloping or curved surfaces. The constant penalty for all larger changes (e.g., regardless of their size) preserves discontinuities. P1 and P2 are referred to as matching-cost smoothing penalties in SGM / eSGM algorithms. As an optimization technique, in some implementations, in addition to storing all path cost values, the minimum path cost of the previous pixels is also stored in an on-chip buffer (e.g., path cost buffer 404) to allow for recalculation of min i L r (pr, i) to avoid.

[0038] In one embodiment, the SGM / eSGM block 206 can use a temporary buffer to store the data from a previous SGM / eSGM pass (e.g., in the path cost buffer 404). The buffer can have the size W × H × dMax, where W is the width of the original stereo image, H is the height of the original stereo image, and dMax is the maximum possible disparity value (e.g., 128 or 256). To reduce the size of the buffer, the SGM / eSGM block 206 can use a buffer whose size is as follows: Buffer size =W×H(pathNum×(bytesPerDisp+costNum×bytesPerCost) +bytesWinnerDisp+bytesWinnerCost) where pathNum is the number of aggregation paths (e.g., 1, 2, or 3), bytesPerDisp is the number of bytes used to represent a disparity value, costNum is the number of costs for subpixel interpolation (e.g., 3), bytesPerCost is the number of bytes used to represent the path cost values, bytesWinnerDisp is the number of bytes used to represent the winning disparity value, and bytesWinnerCost is the number of bytes used to represent the winning path cost value.

[0039] In some embodiments, the SGM / eSGM process of the SGM / eSGM block 206 uses a 3-pass method. An example of a 3-pass method is shown in Fig. Figure 5B illustrates this. Operation “A” shows the first pass, in which the path cost array contains a winning pixel for each of the paths L0, L1, L2, and L3, identified by a shading pattern. The sum of all path cost values ​​is represented by the array “Sp”. Sp represents the winning pixels from each of the four paths and also identifies the pixels adjacent to the winning pixels, since certain calculations may, for example, use neighbor pixel information, as explained below. The following notation is used in the procedures described below: L direction #pass (Pixel position, disparity): Path cost value for the specified direction and passes, indexed by the specified disparity; Sp #paSS (Pixel position, disparity): Subtotal of the path cost values ​​of the 4 directions of the specified pass, indexed by the given disparity; S: aggregated path cost values ​​from all 8 directions; and d direction #pass Winner disparity value along the specified direction for the specified pass

[0040] In some embodiments, the first pass of operation "A" is performed from the upper left corner of the image to the lower right corner. The first pass can have the following for each pixel: Calculating L r 1 (p, d) for the 4 directions; Calculating Sp 1 = L0 1 (p, d0) + L1 1 (p, d1) + L2 1 (p, d2) + L3 1 (p, d3); Selecting the minimum L r 1 (p, d) from the 4 path cost values, where d r 1 the disparity value that corresponds to the selected path cost; and Spending Sp 1 (p, d r 1 ), Sp 1 (p, d r 1 +1), Sp1 (p, d r 1 -1) and d r 1 , where d r 1 +1 and d r 1 -1 the disparity values ​​are those corresponding to the pixels that are attached to the winning pixel (i.e. the pixel that d min 1 (corresponds) adjacent.

[0041] Operation "B" shows the second pass, in which the path cost value ordering is determined for each of the paths L4, L5, L6, and L7, and illustrates the determination of the winning candidates in Operation "C". The sum array from the first pass is summed with the sum array from the second pass to produce an initial winning candidate array. Then, the first winner is selected from the first winning candidate array and subjected to subpixel refinement (explained below) in Operation "D" to generate the first winning disparity.

[0042] In some embodiments, the second pass is performed from the lower right corner to the upper left corner of the image. The second pass can have the following for each pixel: Loading Sp 1 (p, d r 1 ), Sp 1 (p, d r 1 +1), Sp 1 (p, d r 1 -1) and d r 1 ; Calculating L r 2 (p, d) for the 4 directions; Calculating Sp 2 = L4 2 (p, d4) + L5 2 (p, d5) + L6 2 (p, d6) + L7 2 (p, d7) Calculating S(p, d) r 1 ) = Sp 1 (p, d r 1 )+Sp 2 (p, d r 1 ), Calculating S(p, d) r 1 +1) = Sp 1 (p, d r 1 +1)+Sp 2 (p, d r 1 +1); Calculating S(p, d) r 1-1) = Sp 1 (p, d r 1 -1)+Sp 2 (p, d r 1 -1); Choosing the minimum L r 1 oder 2 (p, d) from the 8 path cost values ​​of S(p, d) r 1 ), where d* is the disparity value corresponding to the selected path cost value; Performing a subpixel interpolation on d* using S(p, d) r 1 ), S(p, d r 1 +1) and S(p, d r 1 -1); Selecting the L r 2 (p, d) with the minimum cost value and the disparity value corresponding to the selected path cost value (d r 2 ) corresponds; and Spending Sp 2 (p, d r 2 ), Sp 2 (p, d r 2 +1), Sp 2 (p, d r 2 -1) and d r 2 .

[0043] Operation "E" shows the third path, where the path costs for L0-L3 are determined in the third pass, and the sum of the path costs from the third pass is calculated to obtain winning candidates in Operation "F". Then, a winner is selected from the winners of the third pass and subjected to subpixel refinement to obtain a second winner disparity and a second winner cost. Finally, based on the first winner disparity and first winner cost determined in the second pass, and the second winner disparity and second winner cost determined in the third pass, a final winner is selected in Operation "G".

[0044] The third pass is performed from the upper left corner of the image to the lower right corner. In the third pass, for each pixel: Loading Sp 2 (p, d min 2 ), Sp2 (p, d min 2 +1), Sp 2 (p, d min 2 -1) and d min 2 ; Loading S(p, d*), d*; Calculating L r 3 (p, d) for the 4 directions; Calculating Sp 3 = L0 3 (p, d0) + L1 3 (p, d1) + L2 3 (p, d2) + L3 3 (p, d3); Calculating S(p, d) r 2 ) = Sp 2 (p, d r 2 ) + Sp 3 (p, d r 2 ); Calculating S(p, d) r 2 +1) = Sp 2 (p, d r 2 +1)+Sp 3 (p, d r 2 +1); Calculating S(p, d) r 2 -1) = Sp 2 (p, d r 2 -1)+Sp 3 (p, d r 2 -1); Selecting the minimum value L r 2 oder 3 (p, d) from the 8 path cost values ​​of S(p, d)r 2 ), where d** is the disparity value corresponding to the selected path cost value; Performing a subpixel interpolation on d** using S(p, d) r 2 ), S(p, d r 2 +1) and S(p, d r 2 -1); and Selecting the minimum of S(p, d*) and S(p, d**) and using the corresponding disparity value as the final disparity value.

[0045] In the above operations AE, the path cost calculator 402 can perform one or more steps that yield a path cost value L. r , calculate a subtotal Sp or an aggregate path cost value S. The winner determination block 406 can perform one or more steps that select a path cost value or a disparity value. Outputting the path cost value L rThe subtotal Sp, the aggregated path cost value S, or the disparity value may involve the path cost calculator 402 outputting such data to the profit determination block 406 and / or the path cost buffer 404. Loading the path cost value L r The calculation of the subtotal Sp, the aggregate path cost value S, or the disparity value may involve the path cost calculator 402 receiving such data from the path cost buffer 404. In some embodiments, one or more of the preceding operations may be performed by another component of the SGM / eSGM block 206 (e.g., the profit determination block 406 may calculate the aggregate path cost value S).

[0046] In some embodiments, the SGM / eSGM block 206, as explained above, can implement subpixel interpolation using the post-processing block 408. The SGM / eSGM block 206 can implement equiangular subpixel interpolation. The equiangular subpixel interpolation for a pixel can be determined as follows: SubpixelEL={Sd+1−Sd−12(Sd−Sd−1),Sd+1≤Sd−1Sd+1−Sd−12(Sd−Sd+1),Sd+1>Sd−1 where S d the minimum path cost value and S d+1 and S d-1 Neighbor path cost values, if available. The value of subpixels EL is added to the disparity value that corresponds to one pixel (in the above designations d* or d**).

[0047] In some implementations, the SGM / eSGM block 206 can provide an intermediate stereo image processing output to the AI ​​block 216 for each pixel in the image. The stereo image processing intermediate output can be the final disparity value for the pixel (i.e., d* or d**, as selected by the procedure described above), the one or more path cost values ​​indexed by the one or more candidate disparity values ​​(i.e., S(d0) to S(d7)), and / or the neighbor path cost values ​​indexed by the one or more candidate disparity values ​​(i.e., S(d0+1) to S(d7+1) and S(d0-1) to S(d7-1)). The AI ​​block 216 can use at least one section of the stereo image processing intermediate output as input for the AI ​​model 218.

[0048] Fig. Figure 6 schematically represents an exemplary architecture of AI Model 218 according to some exemplary embodiments. AI Model 218 can incorporate a convolutional neural network (CNN). The CNN, which is a specific type of artificial neural network (ANN), can host multiple layers of convolutional filters. Pooling can be performed, and nonlinearities can be handled in lower layers, on which a multilayer perceptron is typically attached.

[0049] For example, as in Fig. As shown in Figure 6, an input 602 is fed to a first convolution layer 604(A). The input 602 may contain the intermediate output of the stereo image processing. The first convolution layer 604(A) may be a fully connected layer. In some embodiments, the first convolution layer 604(A) may have 32 filters used within this layer 604(A). The first convolution layer 604(A) may use the filters to perform convolution operations and to generate one or more feature maps as output. The output of the first convolution layer 604(A) may be fed to a first ReLU (Rectified Linear Unit) 606(A). The first ReLU 606(A) may have an activation function that outputs the input if the input is greater than 0, or 0 if the input is 0 or negative.

[0050] A second convolution layer 604(B) can receive the output of the first ReLU 606(A). The second convolution layer 604(B) can be the same size as the first convolution layer 604(A) or a different size. The second convolution layer 604(B) can use its filters to perform convolution operations and generate one or more feature maps as output. The output of the second convolution layer 604(B) can be fed to a second ReLU 606(B). The output of the second ReLU 606(B) can be fed to a third convolution layer 604(C), and this process can be repeated for the third convolution layer 604(C) and a third ReLU 606(C). The third convolution layer 604(C) can be the same size as the first and second convolution layers 604(A) and 604(B), or a different size.

[0051] The output of the third ReLU 606(C) can be received by a pooling layer 608. Pooling layer 608 can reduce the dimensions of the input data. For example, pooling layer 608 can reduce the dimensions of the input data from 32 to 8. Pooling layer 608 can provide its output to a fourth ReLU 606(D). The fourth ReLU 606(D) can provide its output to a softmax function 610. Softmax function 610 can convert the output of the fourth ReLU 606(D) into a probability distribution to normalize the output. The output 612 of softmax function 610 can contain one or more weights w0 to w1. n The system contains weights, where n is the number of weights. The one or more weights can have 8 weights (e.g., one weight per candidate disparity value (d0-d7) generated by the SGM / eSGM procedure). The Kl model 218 can provide output 612 to the disparity value calculator 220.

[0052] In one embodiment, the disparity value calculator 220 can be configured to multiply a weight from output 612 by a respective candidate disparity value contained in the intermediate output of the stereo image processing. The disparity value calculator 220 can add these products to calculate the final disparity value for the pixel. For example, if the one or more weights have 8 values, the disparity value calculator 220 can calculate the final disparity value as w0 * d0 + w1 * d1 + ... + w7 * d7.

[0053] Fig. Figure 7 is a flowchart illustrating an exemplary procedure 700 for Kl stereo diparity estimation. In block 702, processing logic generates a cost-volume matrix based on a stereo image pair. Each entry in the cost-volume matrix can correspond to a pixel of a first image of the stereo image pair. The cost-volume matrix can contain the cost-volume matrix 318, as described above in relation to Fig. 3 explained.

[0054] In block 704, the processing logic generates a disparity map for the first image of the stereo image pair. The disparity map can contain a disparity value corresponding to each pixel of the first image. The calculation of the pixel-corresponding disparity value can have one or more subblocks 706-710.

[0055] In block 706, the processing logic performs stereo image processing on the cost volume matrix entry corresponding to the pixel to generate an intermediate stereo image processing output. Performing the stereo image processing may involve SGM / eSGM, as described above. Fig. 4 and Fig. 5A-B explains how to generate the intermediate output of the stereo image processing. The intermediate output of the stereo image processing can contain one or more path cost values, indexed by one or more intermediate disparity values. The one or more path cost values ​​can be S(d0) to S(d7), S(d0+1) to S(d7+1), and / or S(d0-1) to S(d7-1). The intermediate output of the stereo image processing can contain one or more candidate disparity values ​​(e.g., the eight candidate disparity values ​​d0 to d7). The intermediate output of the stereo image processing can contain the minimum disparity value of one or more intermediate disparity values. The minimum disparity value can be, as explained above, the smaller of d* or d**.In one embodiment, the implementation of the stereo image processing can include subpixel refinement using the minimum disparity value and one or more neighboring disparity values, as explained above.

[0056] In block 708, the processing logic, using the intermediate output of the stereo image processing as input 602 for a CNN (e.g., the Kl model 218), generates one or more weighting values, as described above in relation to Fig. 6 explained. For example, the CNN can use as input 602 one or more path cost values ​​(e.g. S(d0) to S(d7), S(d0+1) to S(d7+1) and / or S(d0-1) to S(d7-1), the intermediate disparity values ​​(e.g. d0 to d7) and the minimum disparity value (e.g. d* or d**).

[0057] In block 710, the processing logic for the pixel calculates the disparity value using one or more intermediate disparity values ​​from the intermediate output of the stereo image processing and one or more weighting values, as explained above with reference to the disparity value calculator 220. The disparity value may be included in a disparity map at an entry corresponding to the pixel. In one embodiment, calculating the disparity value for the pixel may involve multiplying each intermediate disparity value of the one or more intermediate disparity values ​​by a corresponding weighting value to generate one or more products, and summing the multiple products to obtain the disparity value for the pixel.As explained above, the intermediate disparity values ​​can, for example, have d0 to d7, the one or more weights can have w0 to w7, and the calculation of the disparity value can have the calculation w0 * d0 + w1 * d1 + ... + w7 * d7.

[0058] Blocks 706-710 can be repeated for each pixel in the first image of the stereo image pair to generate the complete disparity map with the calculated disparity values ​​for each pixel. The System 100 from Fig. 1 or a computing device in data communication with the System 100 can use the disparity map for one or more applications, including object / pedestrian detection / tracking, SFM determination, SLAM, virtual reality applications, gaming applications, deep learning applications, or other applications.

[0059] In some embodiments, Method 700 further includes training the CNN using training data elements. Each training data element can have an intermediate training output from the stereo image processing and a target output, a training disparity value. The intermediate output from the stereo image processing can include path cost values, candidate disparity values, and / or a final disparity value generated from a pair of stereo images designed to provide a predetermined intermediate output from the stereo image processing.Training the CNN using one or more elements of the training data may involve calculating a loss between the calculated disparity value (a disparity value calculated based on the weights output by the CNN in response to receiving the training intermediate output of the stereo image processing) and the training disparity value, and subsequently adjusting one or more weights of the CNN using backward propagation and the loss.

[0060] Other variations are within the spirit of the present disclosure. Thus, while various modifications and alternative constructions can be made with respect to the disclosed methods, certain illustrated embodiments are shown in the drawings and have been described in detail above. However, it is understood that the intention is not to limit the disclosure to the specific disclosed form or forms, but rather, on the contrary, to cover all modifications, alternative constructions, and equivalents that fall within the spirit and scope of the disclosure as defined in the attached claims.

[0061] The use of the terms "a," "an," "the," and similar referents in the context of describing disclosed embodiments (particularly in the context of the following claims) is to be interpreted as covering both the singular and the plural unless otherwise specified herein or the context clearly contradicts this, and not as defining an expression. The terms "comprising," "having," "including," and "containing" are to be interpreted as open expressions (i.e., in the sense of "including without being limited to") unless otherwise specified. "Connected" is to be interpreted as partially or completely contained within, attached to, or joined to one another when used unmodified and referring to physical connections, even if an element is inserted between them.The mention of value ranges herein is intended merely as a quick method of individually referring to each separate value falling within the range, unless otherwise stated herein, and each separate value is included in the description as if it were individually reproduced herein. The use of the term "set" (e.g., "a set of objects") or "subset" in at least one embodiment is to be understood as a non-empty compilation comprising one or more elements, unless otherwise noted or the context contradicts this. Furthermore, unless otherwise stated or the context contradicts this, the term "subset" of a corresponding set does not necessarily mean a proper subset of the corresponding set, but the subset and the corresponding set may be the same.

[0062] Unless specifically stated otherwise or the context clearly contradicts it, connective language, such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C," is otherwise to be understood in the context in which it is generally used to indicate that an object, expression, etc., can be either A, B, or C, or any non-empty subset of the sentence consisting of A, B, and C. For example, in the illustrated example of a sentence containing three elements, the connective phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any one of the following: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}.Thus, such connecting expressions should generally not express that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise stated or contradicted by the context, the term "multiplicity" also denotes a state of plurality (e.g., "a multitude of elements" denotes multiple elements). In at least one embodiment, a multitude consists of at least two elements, but may also include more if this is either explicitly stated or indicated by the context. Furthermore, unless otherwise stated or evident from the context, the phrase "based on" means "at least partially based on" or "at least based on," and not "exclusively based on."

[0063] The operations of processes described herein may be performed in any suitable order, unless otherwise specified herein or the context clearly precludes it. In at least one embodiment, a process, such as the processes described herein (or variations and / or combinations thereof), is carried out under the control of one or more computer systems configured with executable instructions, and is implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, by hardware or combinations thereof. In at least one embodiment, code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions that can be executed by one or more processors.In at least one embodiment, a computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transitory signals (e.g., a propagating transient electrical or electromagnetic transmission) but includes non-transitory data storage circuits (e.g., buffers, caches, and queues) within transient signal senders / receivers. In at least one embodiment, the code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media containing executable instructions (or other storage for executable instructions) which, when executed (i.e., as a result of execution) by one or more processors of a computer system, cause the computer system to perform the operations described herein.In at least one embodiment, a set of nontransitory computer-readable storage media comprises multiple nontransitory computer-readable storage media. One or more of these individual nontransitory storage media do not contain the entire code, while multiple nontransitory computer-readable storage media collectively store the entire code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors—for example, one nontransitory computer-readable storage medium stores instructions, and a main CPU executes some of the instructions, while a GPU executes other instructions.In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

[0064] Accordingly, computer systems in at least one embodiment are configured to implement one or more services that, individually or collectively, perform operations of the processes described herein, and such computer systems are configured with applicable hardware and / or software that enables the execution of operations. Furthermore, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment, a distributed computer system comprising several devices that operate differently, such that the distributed computer system performs the operations described herein and such that a single device does not perform all operations.

[0065] The use of any and all examples or illustrative language (e.g., "such as") provided in this document is intended solely to better clarify the embodiments of the disclosure and does not constitute a limitation of the scope of the disclosure unless claimed otherwise. No wording in the description should be interpreted as indicating any unclaimed element as essential to the implementation of the disclosure.

[0066] Any references, including publications, patent applications and patents mentioned in this document, are hereby incorporated by reference to the same extent as if each reference had been individually and specifically indicated as being included by reference and set forth in this document in its entirety.

[0067] In the description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It is understood that these terms are not intended to be synonymous. Rather, in specific examples, "connected" or "coupled" can be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" can also mean that two or more elements are not in direct contact with each other, but nevertheless interact or work together.

[0068] Unless expressly stated otherwise, in some embodiments, terms such as "processing", "calculating", "calculating", "determining" or the like throughout the description are understood to refer to actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or convert data represented as physical, e.g. electronic, quantities in the registers and / or memories of the computing system into other data represented in a similar manner as physical quantities in the memories, registers or other such information storage, transmission or display devices of the computing system.

[0069] Similarly, the term "processor" can refer to any device or section of a device that processes electronic data from registers and / or memories and converts that electronic data into other electronic data that can be stored in registers and / or memories. As non-restrictive examples, the "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, "software" processes can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Furthermore, each process can refer to multiple processes for executing instructions sequentially or in parallel, continuously or intermittently.The terms “system” and “method” are used interchangeably in this document for at least one embodiment insofar as a system can embody one or more methods and the methods can be considered as a system.

[0070] This document may refer to the acquisition, capture, reception, or input of analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, a method for acquiring, capturing, receiving, or inputting analog and digital data can be accomplished in various ways, such as receiving data as a parameter of a function call or a call to an application programming interface. In some implementations, the process of acquiring, capturing, receiving, or inputting analog or digital data can be achieved by transmitting data over a serial or parallel interface.In another implementation, processes of acquiring, capturing, receiving, or inputting analog or digital data can be accomplished by transmitting data over a computer network from the providing entity to the capturing entity. At least one embodiment can also refer to providing, outputting, transmitting, sending, or displaying analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or displaying analog or digital data can be accomplished by transmitting data as an input or output parameter of a function call, a parameter of an application programming interface, or an interprocess communication mechanism.

[0071] Although the descriptions presented here are exemplary embodiments of the described methods, other architectures may also be used to implement the described functionality, and these are intended to be within the scope of this disclosure. Furthermore, although specific distributions of responsibilities are defined above for the purpose of description, various functions and responsibilities could be distributed and divided differently depending on the circumstances.

[0072] Although the subject matter has been further described in language specific to structural features and / or process steps, it is understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or steps described. Rather, specific features and steps are disclosed as exemplary ways of implementing the claims.

Claims

[1] Procedure encompassing: Generating a cost-volume matrix based on a stereo image pair, where each entry in the cost-volume matrix corresponds to a pixel of a first image of the stereo image pair; and Generating a disparity map for the first image of the stereo image pair, wherein the disparity map includes a disparity value corresponding to each pixel of the first image, and wherein calculating the disparity value corresponding to the pixel includes: Performing stereo image processing with the entry corresponding to the pixel to generate an intermediate output of the stereo image processing, Generating a multitude of weighting values ​​using the intermediate output of stereo image processing as input for a convolutional neural network (CNN), and Calculating the disparity value for the pixel using a variety of intermediate disparity values ​​from the intermediate output of the stereo image processing and a variety of weighting values. [2] Method according to claim 1, wherein the calculation of the disparity value for the pixel comprises: Multiplying each intermediate disparity value of the multitude of intermediate disparity values ​​by a corresponding weighting value to generate a multitude of products; and Summing up the multitude of products as the disparity value for the pixel. [3] Method according to claim 1 or 2, wherein the intermediate output of the stereo image processing further comprises: a multitude of path cost values, indexed by the multitude of intermediate disparity values; and a minimum disparity value among the multitude of intermediate disparity values. [4] Method according to claim 3, wherein generating the plurality of weighting values ​​further comprises using the plurality of path cost values ​​and the minimum disparity value as a further input to the CNN. [5] Method according to claim 3 or 4, wherein performing the stereo image processing further comprises performing subpixel refinement using the minimum disparity value and one or more neighboring disparity values. [6] Method according to any of the preceding claims, wherein the stereo image processing comprises efficient semi-global matching, eSGM. [7] Method according to any one of the preceding claims, wherein the CNN comprises: three folding layers; and a pooling layer. [8] A method according to any of the preceding claims, further comprising training the CNN using a plurality of elements of training data, wherein each element of the training data has an intermediate training output of the stereo image processing and a training disparity value as the target output, and wherein training the CNN using the plurality of elements of training data comprises: Calculating a loss between the calculated disparity value and the training disparity value; and Adjusting one or more weights of the CNN using backward propagation and loss. [9] System encompassing: one or more processing units for performing operations, comprising: Generating a cost-volume matrix based on a stereo image pair, where each entry in the cost-volume matrix corresponds to a pixel of a first image of the stereo image pair; and Generating a disparity map for the first image of the stereo image pair, wherein the disparity map includes a disparity value corresponding to each pixel of the first image, and wherein calculating the disparity value corresponding to the pixel includes: Performing stereo image processing with the entry corresponding to the pixel to generate an intermediate output of the stereo image processing, Generating a multitude of weighting values ​​using the intermediate output of stereo image processing as input for a convolutional neural network (CNN), and Calculating the disparity value for the pixel using a variety of intermediate disparity values ​​from the intermediate output of the stereo image processing and a variety of weighting values. [10] System according to claim 9, wherein calculating the disparity value for the pixel comprises: Multiplying each intermediate disparity value of the multitude of intermediate disparity values ​​by a corresponding weighting value to generate a multitude of products; and Summing up the multitude of products as the disparity value for the pixel. [11] System according to claim 9 or 10, wherein the intermediate output of the stereo image processing further comprises: a multitude of path cost values, indexed by the multitude of intermediate disparity values; and a minimum disparity value among the multitude of intermediate disparity values. [12] System according to claim 11, wherein generating the plurality of weighting values ​​further comprises using the plurality of path cost values ​​and the minimum disparity value as a further input to the CNN. [13] System according to claim 11 or 12, wherein performing the stereo image processing further comprises performing subpixel refinement using the minimum disparity value and one or more neighboring disparity values. [14] System according to any one of claims 9 to 13, wherein the stereo image processing comprises efficient semi-global matching, eSGM. [15] System according to any one of claims 9 to 14, wherein the CNN comprises: three folding layers; and a pooling layer. [16] System according to any one of claims 9 to 15, further comprising training the CNN using a plurality of elements of training data, wherein each element of training data comprises an intermediate training output of the stereo image processing and a training disparity value as a target output, and wherein training the CNN using the plurality of elements of training data comprises: Calculating a loss between the calculated disparity value and the training disparity value; and Adjusting one or more weights of the CNN using backward propagation and loss. [17] Processor comprising one or more processing units, for: Generating a cost-volume matrix based on a stereo image pair, where each entry in the cost-volume matrix corresponds to a pixel of a first image of the stereo image pair; and Generating a disparity map for the first image of the stereo image pair, wherein the disparity map includes a disparity value corresponding to each pixel of the first image, and wherein calculating the disparity value corresponding to the pixel includes: Performing stereo image processing with the entry corresponding to the pixel to generate an intermediate output of the stereo image processing, Generating a multitude of weighting values ​​using the intermediate output of stereo image processing as input for a convolutional neural network (CNN), and Calculating the disparity value for the pixel using a variety of intermediate disparity values ​​from the intermediate output of the stereo image processing and a variety of weighting values. [18] Processor according to claim 17, wherein calculating the disparity value for the pixel comprises: Multiplying each intermediate disparity value of the multitude of intermediate disparity values ​​by a corresponding weighting value to generate a multitude of products; and Summing up the multitude of products as the disparity value for the pixel. [19] Processor according to claim 17 or 18, wherein the intermediate output of the stereo image processing further comprises: a multitude of path cost values, indexed by the multitude of intermediate disparity values; and a minimum disparity value among the multitude of intermediate disparity values. [20] Processor according to claim 19, wherein generating the plurality of weight values ​​further comprises using the plurality of path cost values ​​and the minimum disparity value as a further input for the CNN.

Images