Processor, processing system, processing method and storage medium for disparity data

Abstract

This invention provides a processor, processing system, processing method, and storage medium for disparity data. The processor is configured with a first memory and a second memory, and is configured to: acquire a first feature map and a second feature map indicating the disparity of the original image; slice the first feature map and the second feature map, and store the obtained multiple sets of slice feature data into multiple first memory sub-units of the first memory; calculate the absolute values ​​of the differences between the multiple sets of first slice feature data of the first feature map and the multiple sets of second slice feature data of the corresponding dimensions of the second feature map, and store the absolute values ​​into multiple second memory sub-units of the second memory; and accumulate the stored data in the multiple second memory sub-units to determine the cost space features indicating the disparity of the original image.

Description

Technical Field

[0001] This invention belongs to the field of depth estimation technology, and particularly relates to a processor for disparity data, a system for processing disparity data, a method for processing disparity data, and a computer-readable storage medium. Background Technology

[0002] Binocular depth estimation technology is widely used in mobile devices, autonomous driving systems, and AR / VR / MR devices. It primarily utilizes image information from the left and right eyes for global or semi-global image matching to calculate the binocular disparity, and then calculates the depth information of the current scene based on camera calibration parameters. However, existing global methods consume significant computational and storage resources, resulting in high costs for scenarios with strong real-time requirements. While semi-global solutions consume relatively fewer resources, their depth estimation performance is less than ideal due to the sacrifice of global information.

[0003] To address these issues, existing technologies have proposed improvements using deep learning to balance reducing computational load and improving depth estimation performance by optimizing the cost volume. However, current deep learning algorithms generally require memory access operators such as Gather and Scatter, which are typically not well-suited for processing by neural network processor units (NNUnits), leading to reduced data processing efficiency and real-time performance.

[0004] In order to overcome the above-mentioned defects in the existing technology, there is an urgent need in the field for a parallax data processing technology to reduce the hardware design complexity of NPU, overcome the dependence of deep learning algorithms on memory access operators, and reduce the write-back process of external storage, so as to improve the efficiency and real-time performance of data processing. Summary of the Invention

[0005] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed descriptions that follow.

[0006] To overcome the aforementioned deficiencies in the prior art, this invention provides a parallax data processor, a parallax data processing system, a parallax data processing method, and a computer-readable storage medium. It reduces the hardware design complexity of the NPU by leveraging the heterogeneous computational cost space characteristics of the Digital Signal Processor (DSP), overcomes the dependence of deep learning algorithms on memory access operators by employing basic operators such as ABS, ADD, and SUB, and fully utilizes the on-chip storage resources of the DSP to reduce the write-back process to external storage, thereby improving data processing efficiency and real-time performance.

[0007] Specifically, according to a first aspect of the present invention, the processor providing disparity data is configured with a first memory and a second memory, and is configured to: acquire a first feature map and a second feature map indicating the disparity of the original image. The data volume of the second feature map is configured to be an integer multiple of the data volume of the first feature map according to the step size of the disparity processing. The first feature map and the second feature map are sliced ​​according to at least one dimension, and the obtained multiple sets of slice feature data are stored in multiple first memory sub-units of the first memory. Based on the stored data in the multiple first memory sub-units, the absolute values ​​of the differences between the multiple sets of first slice feature data of the first feature map and the multiple sets of second slice feature data of the corresponding dimension of the second feature map are calculated, and the absolute values ​​are stored in multiple second memory sub-units of the second memory. Furthermore, based on the at least one dimension, the stored data in the multiple second memory sub-units are accumulated, and a cost space feature indicating the disparity of the original image is determined according to the accumulated value of each absolute value.

[0008] Furthermore, in some embodiments of the present invention, the processor is connected to an external static memory, and the step of obtaining the first feature map and the second feature map indicating the parallax of the original image includes: obtaining the first feature map and the second feature map stored therein via the static memory.

[0009] Furthermore, in some embodiments of the present invention, the first feature map and the second feature map respectively include stride feature dimension N, channel feature dimension C, height feature dimension H and width feature dimension W. The step of slicing the first feature map and the second feature map according to at least one dimension of the first feature map and the second feature map includes: obtaining disparity feature data of disparity feature dimension D; and slicing the first feature map and the second feature map into multiple sets of slice feature data according to the disparity feature dimension D and the channel feature dimension C.

[0010] Further, in some embodiments of the present invention, the step of calculating the absolute value of the difference between multiple sets of first slice feature data of the first feature map and multiple sets of second slice feature data of the corresponding dimension of the second feature map based on the stored data in the plurality of first memory sub-units includes: using a first NCHW tensor to perform segmentation processing on the first slice feature data based on the width feature dimension W to obtain multiple sets of first segmentation feature data; using a second NCHW tensor to perform segmentation processing on the second slice feature data based on a preset stride and the width feature dimension W to obtain multiple sets of second segmentation feature data of the second slice feature data under the preset stride; and calculating the absolute value of the difference between each set of first segmentation feature data and its corresponding width feature dimension second segmentation feature data with respect to the channel feature dimension C based on the first NCHW tensor and the second NCHW tensor.

[0011] Further, in some embodiments of the present invention, the step of accumulating the stored data in the plurality of second memory sub-units based on the at least one dimension, and determining the cost space feature indicating the disparity of the original image based on the accumulated value of each absolute value includes: determining the number of iterations based on a preset disparity range and the preset stride; accumulating each absolute value cyclically based on the number of iterations to determine a plurality of third NCHW tensors. Each third NCHW tensor indicates a cost space feature of the disparity of the original image along the corresponding channel feature dimension C; and calculating a fourth NCHW tensor based on the plurality of third NCHW tensors as the cost space feature.

[0012] Further, in some embodiments of the present invention, the step of accumulating the stored data in the plurality of second memory sub-units based on the at least one dimension, and determining the cost space feature indicating the disparity of the original image based on the accumulated value of each absolute value includes: accumulating the stored data in the plurality of second memory sub-units based on the channel feature dimension C to determine the inner loop accumulated value of each absolute value with respect to the channel feature dimension C; storing each inner loop accumulated value in a plurality of third storage sub-units of the static memory; initializing the plurality of second memory sub-units to perform an inner loop accumulated calculation based on the next channel feature dimension C; accumulating the stored data in the plurality of third storage sub-units based on the disparity feature dimension D to determine the outer loop accumulated value of each inner loop accumulated value with respect to the disparity feature dimension D; and determining the cost space feature indicating the disparity of the original image based on each outer loop accumulated value.

[0013] Furthermore, in some embodiments of the present invention, the processor is also connected to an external dynamic memory. The static memory obtains the first feature map and the second feature map from the dynamic memory via data prefetching. The processor is further configured to: while performing the next operation, write back each of the outer loop accumulation values ​​and store them in a plurality of fourth storage sub-units of the dynamic memory, so as to use the operation time of the next operation to mask the write-back and storage write-back time.

[0014] Furthermore, in some embodiments of the present invention, the dynamic memory is further configured with a plurality of fifth storage sub-units, and the processor is also connected to an external neural network processing unit via the dynamic memory. The neural network processing unit is configured to: acquire multiple real-scene images and extract features from them to determine a first feature map and a second feature map; store the first feature map and the second feature map respectively in the plurality of fifth storage sub-units for retrieval by the static memory; acquire the stored outer loop accumulation value via the plurality of fourth storage sub-units of the dynamic memory to determine cost space features indicating the disparity of the original image; and perform cost aggregation on the cost space features to calculate the disparity of the multiple real-scene images.

[0015] Furthermore, in some embodiments of the present invention, the processor described above according to the first aspect of the present invention is further configured with at least one hardened calculation circuit. The hardened calculation circuit is selected from at least one of a subtraction operation circuit, an absolute value operation circuit, an addition operation circuit, and an accumulator circuit.

[0016] Furthermore, the disparity data processing system provided by the second aspect of the present invention includes: a neural network processing unit, configured to extract features from multiple acquired real-world images to determine corresponding first feature maps and second feature maps, and to perform cost aggregation on cost space features determined based on the first feature maps and second feature maps to calculate the disparity of the multiple real-world images; and a processor for disparity data as described in any of the preceding claims provided by the first aspect of the present invention, configured to determine corresponding cost space features based on the first feature maps and second feature maps.

[0017] Furthermore, in some embodiments of the present invention, the disparity data processing system further includes: a static memory connected to the neural network processing unit and the processor, and configured with a plurality of third storage sub-units and / or a plurality of sixth storage sub-units. The plurality of third storage sub-units are used to store the inner loop accumulation value determined by accumulating the stored data in the plurality of second memory sub-units of the processor. The plurality of sixth storage sub-units are used to pre-store the first feature map and the second feature map.

[0018] Furthermore, in some embodiments of the present invention, the parallax data processing system further includes: a dynamic memory connected to the neural network processing unit and the processor, and configured with a plurality of fourth storage sub-units and / or a plurality of fifth storage sub-units, wherein the plurality of fourth storage sub-units are used to store the outer loop accumulation value determined by accumulating the stored data in the plurality of third storage sub-units, and the plurality of fifth storage sub-units are used to store the first feature map and the second feature map extracted by the neural network processing unit.

[0019] Furthermore, the disparity data processing method provided by the third aspect of the present invention includes the following steps: acquiring a first feature map and a second feature map indicating the disparity of the original image. The data volume of the second feature map is configured to be an integer multiple of the data volume of the first feature map according to the stride of the disparity processing. The first feature map and the second feature map are sliced ​​according to at least one dimension, and the obtained multiple sets of sliced ​​feature data are stored in multiple first memory sub-units of the processor's first memory. Based on the stored data in the multiple first memory sub-units, the absolute values ​​of the differences between the multiple sets of first sliced ​​feature data of the first feature map and the multiple sets of second sliced ​​feature data of the corresponding dimension of the second feature map are calculated, and the absolute values ​​are stored in multiple second memory sub-units of the processor's second memory. And based on the at least one dimension, the stored data in the multiple second memory sub-units are accumulated, and a cost space feature indicating the disparity of the original image is determined according to the accumulated value of each absolute value.

[0020] Furthermore, according to a fourth aspect of the present invention, a computer-readable storage medium stores computational instructions thereon. When the computer instructions are executed by a processor, the parallax data processing method as described in the third aspect of the present invention is implemented. Attached Figure Description

[0021] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0022] Figure 1 A schematic diagram of the architecture of a parallax data processing system provided according to some embodiments of the present invention is shown.

[0023] Figure 2 A flowchart illustrating a method for processing parallax data according to some embodiments of the present invention is shown.

[0024] Figure 3A schematic diagram of data interaction is shown for a method of processing parallax data according to some embodiments of the present invention.

[0025] Figure 4 A schematic diagram of data interaction is shown for a method of processing parallax data according to some embodiments of the present invention. Detailed Implementation

[0026] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Although the description of the present invention is presented in conjunction with preferred embodiments, this does not mean that the features of the invention are limited to these embodiments. On the contrary, the purpose of describing the invention in conjunction with embodiments is to cover other options or modifications that may be derived based on the claims of the present invention. To provide a thorough understanding of the invention, many specific details will be included in the following description. The invention may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of the invention, some specific details will be omitted in the description.

[0027] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0028] Furthermore, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood as the orientations shown in the relevant paragraphs and accompanying drawings. These relative terms are for illustrative purposes only and do not imply that the described apparatus must be manufactured or operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0029] It is understood that although terms such as "first," "second," and "third" may be used herein to describe various components, regions, layers, and / or parts, these components, regions, layers, and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers, and / or parts. Therefore, the first components, regions, layers, and / or parts discussed below may be referred to as second components, regions, layers, and / or parts without departing from some embodiments of the present invention.

[0030] As mentioned above, existing deep learning algorithms generally require memory access operators such as Gather and Scatter, but neural network processor units are generally not good at handling these memory access operators, which will reduce their data processing efficiency and real-time performance.

[0031] To overcome the aforementioned deficiencies in the prior art, this invention provides a parallax data processor, a parallax data processing system, a parallax data processing method, and a computer-readable storage medium. It reduces the hardware design complexity of the NPU by leveraging the heterogeneous computational cost space characteristics of the Digital Signal Processor (DSP), overcomes the dependence of deep learning algorithms on memory access operators by employing basic operators such as ABS, ADD, and SUB, and reduces the write-back process to external storage by fully utilizing the on-chip storage resources of the DSP, thereby improving data processing efficiency and real-time performance.

[0032] In some non-limiting embodiments, the parallax data processing method provided in the third aspect of the present invention can be implemented via the parallax data processor provided in the first aspect of the present invention. Specifically, the parallax data processor can be configured for use in the parallax data processing system provided in the second aspect of the present invention, wherein a memory and a processor are configured. The memory includes, but is not limited to, the computer-readable storage medium provided in the fourth aspect of the present invention, on which computer instructions are stored. The processor is connected to the memory and configured to execute the computer instructions stored in the memory to implement the parallax data processing method provided in the third aspect of the present invention.

[0033] Please refer to the following first. Figure 1 , Figure 1 A schematic diagram of the architecture of a parallax data processing system provided according to some embodiments of the present invention is shown.

[0034] like Figure 1 As shown, the disparity data processing system provided according to the second aspect of the present invention can be configured with a neural network processing unit (NPU) and a disparity data processor. Here, the disparity data processor can be a digital signal processor (DSP), which is configured with a first memory and a second memory for caching slice feature data during the computation process, thereby reducing the write-back process to external storage.

[0035] Furthermore, in Figure 1In the illustrated embodiment, the parallax data processing system may also preferably be configured with static RAM and / or dynamic DDR memory. The neural network processing unit (NPU) and digital signal processor (DSP) can be connected via the SRAM and / or DDR memory to cooperate in completing the parallax data processing flow.

[0036] The following will describe the working principle of the above-mentioned processing system and digital signal processor (DSP) with reference to some embodiments of parallax data processing methods. Those skilled in the art will understand that these embodiments of processing methods are merely non-limiting implementations provided by the present invention, intended to clearly demonstrate the main concepts of the invention and provide specific solutions convenient for public implementation, rather than limiting all functions or operating modes of the processing system and DSP. Similarly, the processing system and DSP are also merely non-limiting implementations provided by the present invention, and do not constitute a limitation on the executing entity or execution order of the steps in these processing methods.

[0037] Please refer to the reference. Figures 1-3 . Figure 2 A flowchart illustrating a method for processing parallax data according to some embodiments of the present invention is shown. Figure 3 A schematic diagram of data interaction is shown for a method of processing parallax data according to some embodiments of the present invention.

[0038] like Figures 1-3 As shown, in the process of parallax data processing, the neural network processing unit (NPU) can first acquire multiple real-world images with parallax from the camera module, and then extract the first feature map and the second feature map indicating the parallax of the original image from them via a pre-trained deep neural network (e.g., BackBone).

[0039] Taking a binocular camera system for AR / VR / MR devices as an example, the multiple original images may include the left-eye image captured by the left-eye camera and the right-eye image captured by the right-eye camera. Correspondingly, the first feature map and the second feature map can be the left-eye feature map and the right-eye feature map indicating the binocular disparity of the binocular camera system, respectively. Here, the left-eye feature map and the right-eye feature map can be represented by a multi-channel tensor of the format NCHW, where N is the stride feature dimension, C is the channel feature dimension, H is the height feature dimension, and W is the width feature dimension. The height feature dimension H and the width feature dimension W are generally smaller than the resolution of the original image. The data volume of the right-eye feature map can be configured as an integer multiple of the data volume of the left-eye feature map, depending on the stride of the disparity processing. For example, in an embodiment where N=4, the data volume of the right-eye feature map can be configured to be 4 times the data volume of the left-eye feature map. Since the specific steps of extracting the first feature map and the second feature map indicating the disparity of the original image via the neural network processing unit (NPU) are prior art in this field, they will not be elaborated here.

[0040] Subsequently, the neural network processing unit (NPU) can store the extracted first and second feature maps into dynamic memory (DDR) for extraction when the digital signal processor (DSP) performs heterogeneous calculations of cost volume features.

[0041] Please refer to the details. Figure 4 , Figure 4 A schematic diagram illustrating the data interaction of a parallax data processing method provided according to some embodiments of the present invention is shown. Figure 4 In the illustrated embodiment, the dynamic memory DDR may be further configured with multiple storage sub-units 411 and 412. The neural network processing unit (NPU) can store the extracted first feature map into storage sub-unit 411 of the dynamic memory DDR and store the extracted second feature map into storage sub-unit 412 of the dynamic memory DDR.

[0042] Furthermore, the static RAM can also be configured with multiple storage sub-cells 421 and 422. Before the heterogeneous computing cost space features are defined, the static RAM can perform step S1, which uses a data pre-fetch method to pre-load the first feature map and the second feature map stored in each storage sub-cell 411 and 412 of the dynamic memory DDR into the corresponding storage sub-cells 421 and 422 of the static RAM. This reduces the bandwidth load of the dynamic memory DDR, thereby further improving the efficiency and real-time performance of data processing.

[0043] After that, such as Figures 1-4As shown, in the process of heterogeneous computing cost space features, the digital signal processor (DSP) can preferably execute step S2, which involves obtaining the first feature map and the second feature map stored in each storage sub-unit 421 and 422 of the static memory (SRAM), and then slicing the first feature map and the second feature map according to at least one dimension of the first feature map and the second feature map, and storing the obtained multiple sets of slice feature data Tile Left and Tile Right into multiple first memory sub-units of the first memory TCM0 in the DSP.

[0044] Specifically, in some embodiments, the first feature map and the second feature map respectively include four-dimensional feature data: stride feature dimension N, channel feature dimension C, height feature dimension H, and width feature dimension W. Here, for an embodiment where the original image height (H) * width (W) is 640 * 480, disparity (D) is 320, and stride is 4, the feature dimension of the first feature map is 1 * 16 * 120 * 160, and the feature dimension of the second feature map is 1 * 16 * 120 * 640.

[0045] The digital signal processor (DSP) can first acquire the disparity feature data of the disparity feature dimension D, and then perform tiling processing on the first feature map according to the disparity feature dimension D and the channel feature dimension C, dividing the first feature map into multiple sets of tiling feature data, and then storing one set of the first tiling feature data into the Tile Left memory subunit of the first memory TCM0.

[0046] Furthermore, the digital signal processor (DSP) can also perform tile processing on the second feature map according to the parallax feature dimension D and the channel feature dimension C, dividing the second feature map into multiple sets of tile feature data, and storing one set of second tile feature data into the Tile Right memory sub-unit of the first memory TCM0. Here, for the embodiment with a stride of 4, the size of the Tile Right memory sub-unit is 4 times that of the Tile Left memory sub-unit.

[0047] Subsequently, the digital signal processor (DSP) can calculate the absolute value of the difference between multiple sets of first slice feature data Tile Left of the first feature map and multiple sets of second slice feature data Tile Right of the corresponding dimension of the second feature map based on the stored data in multiple first memory sub-units of the first memory TCM0, and store the absolute value in multiple second memory sub-units of the second memory TCM1. Then, it can accumulate the stored data in multiple second memory sub-units based on at least one dimension to determine the cost volume feature indicating the disparity of the original image.

[0048] Specifically, such as Figure 4 As shown, the digital signal processor (DSP) can first prepare four second memory sub-units Acc0 to Acc3 in the second memory TCM1, and then perform interleaving (Interleave) processing on the second feature map according to the width feature dimension W to transform it into four intermediate feature maps R Shuffle0 to R Shuffle3 of the same size as the first feature map.

[0049] Subsequently, the digital signal processor (DSP) can use the SUB operator to subtract each intermediate feature map from the first feature map to obtain the corresponding difference value, and then store the difference value into the corresponding second memory sub-units Acc0 to Acc3.

[0050] Furthermore, to avoid the introduced disparity feature data escalating the cost space operation to a five-dimensional feature operation, the digital signal processor (DSP) can pre-slice the second feature map. The second NCHW tensor is used to segment the second slice feature data based on a preset stride and width feature dimension W, thereby obtaining multiple sets of second segmented feature data under the preset stride.

[0051] right_slice_0=right[:,:,:,0:W:4]

[0052] right_slice_1=right[:,:,:,1:W:4]

[0053] right_slice_2=right[:,:,:,2:W:4]

[0054] right_slice_3=right[:,:,:,3:W:4].

[0055] Furthermore, the digital signal processor (DSP) can also perform slice processing on the first feature map, using the first NCHW tensor to perform slice processing on the first slice feature data based on the width feature dimension W, to obtain multiple sets of first segmented feature data:

[0056] left_slice=left[:,:,:,i:].

[0057] In this way, the digital signal processor (DSP) can calculate the difference between each group of first segmentation feature data and its corresponding width feature dimension second segmentation feature data with respect to the channel feature dimension C, based on the first NCHW tensor and the second NCHW tensor.

[0058] Next, the digital signal processor (DSP) can execute step S3, using the ABS operator to perform absolute value operations on the difference data stored in each of the second memory sub-units Acc0 to Acc3, and then execute step S4, using the ADD operator to perform an accumulation operation on the channel feature dimension C of the absolute value data stored in each of the second memory sub-units Acc0 to Acc3, to determine the accumulated value of each absolute value, i.e.

[0059] volume[:,i*4+0,:,:]=Sum(Abs(left_slice-right_slice_0[:,:,:,0:Wi])

[0060] volume[:,i*4+1,:,:]=Sum(Abs(left_slice-right_slice_1[:,:,:,0:Wi])

[0061] volume[:,i*4+2,:,:]=Sum(Abs(left_slice-right_slice_2[:,:,:,0:Wi])

[0062] volume[:,i*4+3,:,:]=Sum(Abs(left_slice-right_slice_3[:,:,:,0:Wi])

[0063] Here, Sum represents the accumulation operation on the channel feature dimension C, and Abs represents the absolute value operation. The result of the operation indicates a cost space feature of the original image disparity along the corresponding channel feature dimension C.

[0064] Next, the digital signal processor (DSP) can determine the number of inner loops (i.e., Loop for Channel = 16) based on the number of channels (C = 16), and determine the number of outer loops (i.e., Loop for Disparity = 320 / 4 = 80) based on the preset disparity range (i.e., D = 320) and preset stride (i.e., Stride = 4). Then, it can perform cumulative calculations based on the number of inner loops and the number of outer loops to determine the cost space characteristics of the above disparity feature data.

[0065] Furthermore, in Figure 4In the illustrated embodiment, the dynamic memory (DDR) may preferably be configured with multiple storage sub-cells 413, and the static memory (SRAM) may preferably be configured with multiple storage sub-cells 423. During the calculation of the aforementioned cost space characteristics, the digital signal processor (DSP) may also preferably combine the storage sub-cells 413 and / or 423 configured in the DDR and / or SRAM to further reduce the write-back process between the DSP and the DDR, thereby further improving data processing efficiency and real-time performance.

[0066] Specifically, in the process of determining the cost space characteristics, the digital signal processor (DSP) can first use the ADD operator to perform cumulative calculations based on the channel feature dimension C on the stored data in multiple second memory sub-units Acc0 to Acc3, so as to determine the inner loop cumulative value of each absolute value with respect to the channel feature dimension C.

[0067] After completing the inner loop accumulation operation of one channel feature dimension C, the digital signal processor (DSP) can store the calculated inner loop accumulation value into multiple storage sub-units 423 of the static memory (SRAM) and initialize each second memory sub-unit Acc0 to Acc3 to perform the inner loop accumulation calculation based on the next channel feature dimension C.

[0068] Then, the digital signal processor (DSP) can also perform outer loop accumulation calculation on the stored data in each storage sub-unit 423 based on the disparity feature dimension D, and determine the outer loop accumulation value of the disparity feature dimension D based on the outer loop accumulation calculation result of each channel feature dimension C in a disparity feature dimension D.

[0069] After completing the outer loop accumulation operation of one disparity feature dimension D, the digital signal processor (DSP) can execute step S5 to store the outer loop accumulation value into multiple storage sub-units 413 of the dynamic memory (DDR), and initialize each storage sub-unit 423 as described above to perform the outer loop accumulation calculation based on the next disparity feature dimension D, until the outer loop accumulation operation of all disparity feature dimensions D is completed.

[0070] Thus, after completing the outer loop accumulation operation of each disparity feature dimension D, the digital signal processor (DSP) can determine the cost space features of the original image disparity based on the outer loop accumulation values ​​stored in each storage sub-unit 413 of the dynamic memory (DDR). This fully utilizes the on-chip storage resources of the DSP and the bandwidth of the dynamic memory (DDR) to further improve the overall data processing efficiency and real-time performance of the disparity data processing system.

[0071] Furthermore, in some embodiments, the digital signal processor (DSP) can preferably write back and store the accumulated values ​​of each outer loop into multiple storage sub-units 413 of the dynamic memory (DDR) while performing the next operation. In this way, the present invention can mask the time spent writing back this data within the next operation, thereby fully utilizing the bandwidth of the DDR to further improve the overall data processing efficiency and real-time performance of the parallax data processing system.

[0072] Furthermore, in some embodiments, the above-mentioned difference operation, accumulation operation and absolute value operation can preferably be implemented by at least one hardened calculation circuit, such as a subtraction operation circuit, an absolute value operation circuit, an addition operation circuit, or an accumulator circuit, configured in the digital signal processor DSP, thereby further reducing the software processing load of the digital signal processor DSP.

[0073] Therefore, compared to existing technologies that first add disparity data to the four-dimensional tensor of NCHW, and then calculate the cost space based on the five-dimensional NCDHW tensor and memory access operators such as Gather and Scatter, this invention, on the one hand, exports the computation process of calculating the cost space to a digital signal processor (DSP) for heterogeneous computation, thereby reducing the hardware design complexity of the NPU; on the other hand, it merges the D-dimensional features into the C-dimensional features, and then uses basic operators such as ABS, ADD, and SUB to realize the calculation function of the cost space, thereby overcoming the dependence of existing deep learning algorithms on memory access operators, and directly outputting the four-dimensional cost space features, thus significantly reducing the data computation load and the amount of output data in the cost space operation process.

[0074] In addition, such as Figure 1 and Figure 3 As shown, after completing the accumulation operation of the outer loop, the neural network processing unit (NPU) can also obtain the stored outer loop accumulation value through multiple storage sub-units 413 of the dynamic memory (DDR) to determine the NCHW four-dimensional cost space features indicating the disparity of the original image, and perform cost aggregation on the four-dimensional cost space features to calculate the disparity of the aforementioned multiple real-scene images. Since the specific scheme for calculating the disparity of multiple real-scene images by cost aggregation of the cost space features via the neural network processing unit (NPU) is prior art in this field, it will not be elaborated upon here.

[0075] In summary, the parallax data processor, parallax data processing system, parallax data processing method, and computer-readable storage medium provided by this invention can reduce the hardware design complexity of the NPU, decrease the amount of output data, and reduce the bandwidth pressure on external storage by heterogeneously computing cost space characteristics on the Digital Signal Processor (DSP). Furthermore, by employing basic operators such as ABS, ADD, and SUB for cost space operations, this invention can overcome the dependence of deep learning algorithms on memory access operators such as Gather or Scatter, thereby improving the MAC utilization rate of the neural network processing unit (NPU). Moreover, by configuring a first memory TCM0 and a second memory TCM1 on the DSP, configuring multiple storage sub-units 411-413 on the dynamic memory DDR, configuring multiple storage sub-units 421-423 on the static memory SRAM, and configuring pre-fetch and layer-by-layer cache accumulation data transmission methods, this invention can also fully utilize the on-chip storage resources of the DSP to reduce the write-back process to external storage, thereby improving data processing efficiency and real-time performance.

[0076] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0077] Those skilled in the art will understand that information, signals, and data can be represented using any of a variety of different techniques and arts. For example, the data, instructions, commands, information, signals, bits, symbols, and chips described throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0078] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0079] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0080] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0081] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A processor for parallax data, characterized in that, The processor is equipped with a first memory and a second memory, and is configured as follows: Obtain a first feature map and a second feature map indicating the disparity of the original image. The first feature map and the second feature map respectively include stride feature dimension N, channel feature dimension C, height feature dimension H and width feature dimension W. The data volume of the second feature map is configured to be an integer multiple of the data volume of the first feature map according to the stride of the disparity processing. Obtain disparity feature data in disparity feature dimension D; The disparity feature data is merged into the channel feature dimension C, and the first feature map and the second feature map are respectively divided into multiple sets of slice feature data; The obtained multiple sets of slice feature data are stored in multiple first memory sub-units of the first memory, respectively; Based on the stored data in the plurality of first memory sub-units, the absolute values ​​of the differences between the plurality of sets of first slice feature data of the first feature map and the plurality of sets of second slice feature data of the corresponding dimensions of the second feature map are calculated respectively, and the absolute values ​​are stored in the plurality of second memory sub-units of the second memory; and Based on at least one dimension of the first feature map and the second feature map, the stored data in the plurality of second memory sub-units are accumulated and calculated, and the cost space feature indicating the disparity of the original image is determined according to the accumulated value of each absolute value, wherein the cost space feature is an NCHW four-dimensional cost space feature.

2. The processor as described in claim 1, characterized in that, The processor is connected to an external static memory, and the step of acquiring the first feature map and the second feature map indicating the disparity of the original image includes: The first feature map and the second feature map stored therein are obtained through the static memory.

3. The processor as described in claim 1, characterized in that, The step of calculating the absolute value of the difference between multiple sets of first slice feature data of the first feature map and multiple sets of second slice feature data of the corresponding dimension of the second feature map based on the stored data in the plurality of first memory sub-units includes: Using the first NCHW tensor, the first slice feature data is segmented based on the width feature dimension W to obtain multiple sets of first segmentation feature data; Using a second NCHW tensor, the second slice feature data is segmented based on a preset stride and the width feature dimension W to obtain multiple sets of second segmentation feature data under the preset stride; and Based on the first NCHW tensor and the second NCHW tensor, calculate the absolute value of the difference between the first segmentation feature data and the second segmentation feature data of the corresponding width feature dimension of each group with respect to the channel feature dimension C.

4. The processor as described in claim 3, characterized in that, The step of accumulating the stored data in the plurality of second memory sub-units based on the at least one dimension, and determining the cost space feature indicating the disparity of the original image based on the accumulated value of each absolute value, includes: The number of cycles is determined based on the preset parallax range and the preset stride. Based on the number of iterations, the absolute values ​​are cyclically accumulated to determine multiple third NCHW tensors, wherein each third NCHW tensor indicates a cost space feature of the original image disparity along the corresponding channel feature dimension C; and A fourth NCHW tensor is calculated based on the plurality of third NCHW tensors to serve as the cost space feature.

5. The processor as described in claim 2, characterized in that, The step of accumulating the stored data in the plurality of second memory sub-units based on the at least one dimension, and determining the cost space feature indicating the disparity of the original image based on the accumulated value of each absolute value, includes: Based on the channel feature dimension C, the stored data in the plurality of second memory sub-units are accumulated and calculated to determine the inner loop accumulation value of each absolute value with respect to the channel feature dimension C. The accumulated values ​​of each inner loop are stored in multiple third storage sub-units of the static memory; Initialize the plurality of second memory sub-units to perform inner loop accumulation calculation based on the feature dimension C of the next channel; Based on the disparity feature dimension D, the stored data in the plurality of third storage sub-units are accumulated to determine the outer loop accumulated value of each inner loop accumulated value with respect to the disparity feature dimension D; and Based on the accumulated values ​​of each outer loop, the cost space features indicating the disparity of the original image are determined.

6. The processor as described in claim 5, characterized in that, The processor is also connected to an external dynamic memory, wherein the static memory obtains the first feature map and the second feature map from the dynamic memory via data prefetching, and the processor is further configured to: While performing the next operation, the accumulated values ​​of each outer loop are written back and stored in multiple fourth storage sub-units of the dynamic memory, so as to use the operation time of the next operation to mask the write-back and storage write-back time.

7. The processor as described in claim 6, characterized in that, The dynamic memory is further configured with multiple fifth storage sub-units, and the processor is also connected to an external neural network processing unit via the dynamic memory, wherein the neural network processing unit is configured as follows: Multiple real-scene images are acquired and their features are extracted to determine the first feature map and the second feature map; The first feature map and the second feature map are respectively stored in the plurality of fifth storage sub-units for use by the static memory; The outer loop accumulation value stored in the plurality of fourth storage sub-units of the dynamic memory is obtained to determine the cost space features indicating the disparity of the original image; and Cost aggregation is performed on the cost space features to calculate the disparity of the multiple real-scene images.

8. The processor as described in any one of claims 1 to 7, characterized in that, The processor is also equipped with at least one hardened calculation circuit, wherein the hardened calculation circuit is selected from at least one of the subtraction operation circuit, absolute value operation circuit, addition operation circuit, and accumulator circuit.

9. A parallax data processing system, characterized in that, include: The neural network processing unit is used to extract features from multiple real-scene images to determine the corresponding first feature map and second feature map, and to perform cost aggregation on the cost space features determined based on the first feature map and second feature map to calculate the disparity of the multiple real-scene images. as well as The processor of disparity data as described in any one of claims 1 to 8 is used to determine the corresponding cost space features based on the first feature map and the second feature map.

10. The parallax data processing system as described in claim 9, characterized in that, Also includes: A static memory, connected to the neural network processing unit and the processor, is configured with a plurality of third storage sub-units and / or a plurality of sixth storage sub-units, wherein the plurality of third storage sub-units are used to store the inner loop accumulation value determined by accumulating the stored data in the plurality of second memory sub-units of the processor, and the plurality of sixth storage sub-units are used to pre-store the first feature map and the second feature map.

11. The parallax data processing system as described in claim 10, characterized in that, Also includes: A dynamic memory, connected to the neural network processing unit and the processor, is configured with a plurality of fourth storage sub-units and / or a plurality of fifth storage sub-units, wherein the plurality of fourth storage sub-units are used to store the outer loop accumulation value determined by accumulating the stored data in the plurality of third storage sub-units, and the plurality of fifth storage sub-units are used to store the first feature map and the second feature map extracted by the neural network processing unit.

12. A method for processing parallax data, characterized in that, Includes the following steps: Obtain a first feature map and a second feature map indicating the disparity of the original image. The first feature map and the second feature map respectively include stride feature dimension N, channel feature dimension C, height feature dimension H and width feature dimension W. The data volume of the second feature map is configured to be an integer multiple of the data volume of the first feature map according to the stride of the disparity processing. Obtain disparity feature data in disparity feature dimension D; The disparity feature data is merged into the channel feature dimension C, and the first feature map and the second feature map are respectively divided into multiple sets of slice feature data; The obtained multiple sets of slice feature data are stored in multiple first memory sub-units of the processor's first memory, respectively; Based on the stored data in the plurality of first memory sub-units, the absolute values ​​of the differences between the plurality of sets of first slice feature data of the first feature map and the plurality of sets of second slice feature data of the corresponding dimensions of the second feature map are calculated respectively, and the absolute values ​​are stored in the plurality of second memory sub-units of the processor's second memory; and Based on at least one dimension of the first feature map and the second feature map, the stored data in the plurality of second memory sub-units are accumulated and calculated, and the cost space feature indicating the disparity of the original image is determined according to the accumulated value of each absolute value, wherein the cost space feature is an NCHW four-dimensional cost space feature.

13. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, the parallax data processing method as described in claim 12 is implemented.

Images