Parallax data processor, parallax data processing system, parallax data processing method and storage medium
By performing heterogeneous calculation cost space feature processing in DSP, using basic operators to replace memory fetch operators, the problems of NPU hardware complexity and low data processing efficiency are solved, and efficient and real-time improvement of parallax data processing is achieved .
Patent Information
- Application Number
- CN202310547124.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2043-05-12
Smart Images

Figure CN118941607A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of depth estimation, and in particular relates to a disparity data processor, a disparity data processing system, a disparity data processing method, and a computer-readable storage medium. Background Art
[0002] Binocular depth estimation technology is widely used in mobile devices, autonomous driving systems, and AR / VR / MR devices. It mainly uses the image information of the left and right eyes for global or semi-global image matching to calculate the binocular disparity, and then calculates the depth information of the current scene based on the camera calibration parameters. However, the existing global method consumes a lot of computing and storage resources, and the cost is very high for scenes with strong real-time requirements. Although the semi-global solution consumes relatively few resources, the depth estimation effect is not ideal due to the sacrifice of global information.
[0003] To this end, the existing technology has proposed an improvement plan to use deep learning technology to compensate for the above problems, and to optimize its cost volume to achieve the goals of reducing the amount of calculation and improving the depth estimation effect. However, the existing deep learning algorithms generally require the use of memory access operators such as Gather and Scatter, while the neural network processor unit is generally not good at processing these memory access operators, which will reduce its data processing efficiency and real-time performance.
[0004] In order to overcome the above-mentioned defects of the prior art, the field is in urgent need of a disparity data processing technology to reduce the hardware design complexity of the NPU, overcome the dependence of deep learning algorithms on memory access operators, and reduce the write-back process of external storage to improve the efficiency and real-time performance of data processing. Summary of the invention
[0005] A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or decisive elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.
[0006] In order to overcome the above-mentioned defects of the prior art, the present invention provides a disparity data processor, a disparity data processing system, a disparity data processing method, and a computer-readable storage medium, which can reduce the hardware design complexity of NPU by heterogeneously calculating cost space features in a digital signal processor (DSP), adopt basic operators such as ABS, ADD, SUB to overcome the dependence of deep learning algorithms on memory access operators, and make full use of DSP on-chip storage resources to reduce the write-back process of external storage, thereby improving the efficiency and real-time performance of data processing.
[0007] Specifically, the processor for providing disparity data according to the first aspect of the present invention is configured with a first memory and a second memory, and is configured to: obtain a first feature map and a second feature map indicating the disparity of the original image. The data volume of the second feature map is configured as an integer multiple of the data volume of the first feature map according to the stride of the disparity processing. According to at least one dimension of the first feature map and the second feature map, the first feature map and the second feature map are sliced, and the obtained multiple groups of slice feature data are respectively stored in multiple first memory sub-units of the first memory. Based on the stored data in the multiple first memory sub-units, the absolute values of the differences between the multiple groups of first slice feature data of the first feature map and the multiple groups of second slice feature data of the corresponding dimensions of the second feature map are respectively calculated, and the absolute values are stored in multiple second memory sub-units of the second memory. And based on the at least one dimension, the stored data in the multiple second memory sub-units are cumulatively calculated, and the cost space feature indicating the disparity of the original image is determined according to the cumulative value of each of the absolute values.
[0008] Furthermore, in some embodiments of the present invention, the processor is connected to an external static memory, and the step of obtaining the first feature map and the second feature map indicating the disparity of the original image includes: obtaining the first feature map and the second feature map stored in the static memory via the static memory.
[0009] Furthermore, in some embodiments of the present invention, the first feature map and the second feature map respectively include a stride feature dimension N, a channel feature dimension C, a height feature dimension H and a width feature dimension W, and the step of slicing the first feature map and the second feature map according to at least one dimension of the first feature map and the second feature map includes: obtaining disparity feature data of a disparity feature dimension D; and according to the disparity feature dimension D and the channel feature dimension C, dividing the first feature map and the second feature map into multiple groups of slice feature data, respectively.
[0010] Further, in some embodiments of the present invention, the step of respectively calculating the absolute values of the differences between multiple groups of first slice feature data of the first feature map and multiple groups of second slice feature data of the corresponding dimensions of the second feature map based on the stored data in the multiple first memory sub-units includes: using a first NCHW tensor to perform segmentation processing on the first slice feature data based on the width feature dimension W to obtain multiple groups of first segmentation feature data; using a second NCHW tensor to perform segmentation processing on the second slice feature data based on a preset stride and the width feature dimension W to obtain multiple groups of second segmentation feature data of the second slice feature data under the preset stride; and based on the first NCHW tensor and the second NCHW tensor, respectively calculating the absolute values of the differences between each group of the first segmentation feature data and the second segmentation feature data of the corresponding width feature dimension with respect to the channel feature dimension C.
[0011] Further, in some embodiments of the present invention, the step of accumulating the stored data in the plurality of second memory subunits based on the at least one dimension and determining the cost space feature indicating the original image disparity according to the accumulated value of each absolute value includes: determining the number of loops according to a preset disparity range and the preset stride; accumulating each absolute value according to the number of loops respectively to determine a plurality of third NCHW tensors. Each of the third NCHW tensors indicates a cost space feature of the original image disparity along the corresponding channel feature dimension C; and calculating a fourth NCHW tensor according to the plurality of third NCHW tensors as the cost space feature.
[0012] Further, in some embodiments of the present invention, the step of performing cumulative calculation on the stored data in the multiple second memory sub-units based on the at least one dimension, and determining the cost space feature indicating the disparity of the original image according to the cumulative value of each absolute value includes: performing cumulative calculation on the stored data in the multiple second memory sub-units based on the channel feature dimension C to respectively determine the inner loop cumulative value of each absolute value with respect to the channel feature dimension C; storing each of the inner loop cumulative values to the multiple third storage sub-units of the static memory; initializing the multiple second memory sub-units to perform inner loop cumulative calculation based on the next channel feature dimension C; performing cumulative calculation on the stored data in the multiple third storage sub-units based on the disparity feature dimension D to respectively determine the outer loop cumulative value of each of the inner loop cumulative values with respect to the disparity feature dimension D; and determining the cost space feature indicating the disparity of the original image according to each of the outer loop cumulative values.
[0013] Furthermore, in some embodiments of the present invention, the processor is also connected to an external dynamic memory. The static memory obtains the first characteristic graph and the second characteristic graph from the dynamic memory by means of data pre-fetching, and the processor is further configured to: while performing the next operation, write back and store the accumulated values of each outer loop to multiple fourth storage subunits of the dynamic memory, so as to use the operation time of the next operation to cover the write-back and storage write-back time.
[0014] Furthermore, in some embodiments of the present invention, the dynamic memory is further configured with a plurality of fifth storage subunits, and the processor is further connected to an external neural network processing unit via the dynamic memory. The neural network processing unit is configured to: obtain a plurality of real scene images, and perform feature extraction on them to determine the first feature map and the second feature map; store the first feature map and the second feature map in the plurality of fifth storage subunits respectively for the static memory to call; obtain the outer loop accumulation values stored in the plurality of fourth storage subunits of the dynamic memory through the plurality of fourth storage subunits to determine the cost space features indicating the disparity of the original image; and perform cost aggregation on the cost space features to calculate the disparity of the plurality of real scene images.
[0015] Further, in some embodiments of the present invention, the processor provided according to the first aspect of the present invention is further configured with at least one hardened calculation circuit, which is selected from at least one of a subtraction operation circuit, an absolute value operation circuit, an addition operation circuit, and an accumulator circuit.
[0016] In addition, the above-mentioned disparity data processing system provided according to the second aspect of the present invention includes: a neural network processing unit, which is used to extract features from the acquired multiple real-scene images to determine the corresponding first feature map and the second feature map, and to perform cost aggregation on the cost space features determined based on the first feature map and the second feature map to calculate the disparity of the multiple real-scene images; and a processor of the disparity data as described in any one of the above items provided by the first aspect of the present invention, which is used to determine the corresponding cost space features based on the first feature map and the second feature map.
[0017] Furthermore, in some embodiments of the present invention, the disparity data processing system further includes: a static memory connected to the neural network processing unit and the processor, and configured with a plurality of third storage subunits and / or a plurality of sixth storage subunits. The plurality of third storage subunits are used to store inner loop cumulative values determined by accumulating the storage data in the plurality of second memory subunits of the processor. The plurality of sixth storage subunits are used to pre-access the first feature map and the second feature map.
[0018] Furthermore, in some embodiments of the present invention, the disparity data processing system also includes: a dynamic memory, connected to the neural network processing unit and the processor, and configured with multiple fourth storage subunits and / or multiple fifth storage subunits, wherein the multiple fourth storage subunits are used to store outer loop cumulative values determined by adding up the storage data in the multiple third storage subunits, and the multiple fifth storage subunits are used to store the first feature map and the second feature map extracted by the neural network processing unit.
[0019] In addition, the above-mentioned method for processing disparity data provided according to the third aspect of the present invention includes the following steps: obtaining a first feature map and a second feature map indicating the disparity of the original image. The data volume of the second feature map is configured as an integer multiple of the data volume of the first feature map according to the stride of the disparity processing. According to at least one dimension of the first feature map and the second feature map, the first feature map and the second feature map are sliced, and the obtained multiple groups of slice feature data are respectively stored in multiple first memory subunits of the first memory of the processor. Based on the stored data in the multiple first memory subunits, the absolute values of the differences between the multiple groups of first slice feature data of the first feature map and the multiple groups of second slice feature data of the corresponding dimensions of the second feature map are respectively calculated, and the absolute values are stored in multiple second memory subunits of the second memory of the processor. And based on the at least one dimension, the stored data in the multiple second memory subunits are cumulatively calculated, and the cost space feature indicating the disparity of the original image is determined according to the cumulative value of each of the absolute values.
[0020] In addition, the computer-readable storage medium provided according to the fourth aspect of the present invention stores computing instructions, and when the computer instructions are executed by a processor, the method for processing disparity data provided according to the third aspect of the present invention is implemented. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar reference numerals.
[0022] Figure 1 A schematic diagram of the architecture of a system for processing disparity data provided according to some embodiments of the present invention is shown.
[0023] Figure 2 A schematic flow chart of a method for processing disparity data provided according to some embodiments of the present invention is shown.
[0024] Figure 3A data interaction diagram of a method for processing disparity data provided according to some embodiments of the present invention is shown.
[0025] Figure 4 A data interaction diagram of a method for processing disparity data provided according to some embodiments of the present invention is shown. DETAILED DESCRIPTION
[0026] The following specific embodiments illustrate the implementation of the present invention, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description will include many specific details. The present invention can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present invention, some specific details will be omitted in the description.
[0027] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0028] In addition, the terms "upper", "lower", "left", "right", "top", "bottom", "horizontal" and "vertical" used in the following description should be understood as the directions shown in the paragraph and the related drawings. Such relative terms are only used for the convenience of description and do not mean that the device described therein must be manufactured or operated in a specific direction, and therefore should not be understood as limiting the present invention.
[0029] It is understood that although the terms "first", "second", "third", etc. may be used herein to describe various components, regions, layers and / or parts, these components, regions, layers and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and / or parts. Therefore, the first component, region, layer and / or part discussed below may be referred to as a second component, region, layer and / or part without departing from some embodiments of the present invention.
[0030] As mentioned above, existing deep learning algorithms generally require the use of memory access operators such as Gather and Scatter, but neural network processor units (Neural Processor Units) are generally not good at processing these memory access operators, which will reduce their data processing efficiency and real-time performance.
[0031] In order to overcome the above-mentioned defects of the prior art, the present invention provides a disparity data processor, a disparity data processing system, a disparity data processing method, and a computer-readable storage medium, which can reduce the hardware design complexity of NPU by heterogeneously calculating cost space features in a digital signal processor (DSP), adopt basic operators such as ABS, ADD, SUB to overcome the dependence of deep learning algorithms on memory access operators, and reduce the write-back process of external storage by making full use of DSP on-chip storage resources, thereby improving the efficiency and real-time performance of data processing.
[0032] In some non-limiting embodiments, the above-mentioned method for processing disparity data provided in the third aspect of the present invention can be implemented via the disparity data processor provided in the first aspect of the present invention. Specifically, the disparity data processor can be configured for use in the above-mentioned disparity data processing system provided in the second aspect of the present invention, which is configured with a memory and a processor. The memory includes but is not limited to the above-mentioned computer-readable storage medium provided in the fourth aspect of the present invention, on which computer instructions are stored. The processor is connected to the memory and is configured to execute the computer instructions stored on the memory to implement the above-mentioned method for processing disparity data provided in the third aspect of the present invention.
[0033] First, please refer to Figure 1 , Figure 1 A schematic diagram of the architecture of a system for processing disparity data provided according to some embodiments of the present invention is shown.
[0034] like Figure 1 As shown, the above-mentioned disparity data processing system provided according to the second aspect of the present invention may be configured with a neural network processing unit NPU and a disparity data processor. Here, the disparity data processor may be a digital signal processor DSP, which is configured with a first memory and a second memory for caching the slice feature data during the operation process to reduce the write-back process of the external storage.
[0035] Further, in Figure 1In the illustrated embodiment, the disparity data processing system may also be preferably configured with a static memory SRAM and / or a dynamic memory DDR. The neural network processing unit NPU and the digital signal processor DSP may be connected via the static memory SRAM and / or the dynamic memory DDR, thereby cooperating with each other to complete the disparity data processing flow.
[0036] The working principle of the above-mentioned processing system and digital signal processor DSP will be described below in conjunction with some embodiments of the parallax data processing method. Those skilled in the art will understand that the embodiments of these processing methods are only some non-restrictive implementation methods provided by the present invention, and are intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are convenient for the public to implement, rather than to limit all functions or all working modes of the processing system and digital signal processor DSP. Similarly, the processing system and digital signal processor DSP are also only some non-restrictive implementation methods provided by the present invention, and do not constitute a limitation on the execution subject or execution order of each step in these processing methods.
[0037] Please refer to Figure 1 to Figure 3 . Figure 2 A schematic flow chart of a method for processing disparity data provided according to some embodiments of the present invention is shown. Figure 3 A data interaction diagram of a method for processing disparity data provided according to some embodiments of the present invention is shown.
[0038] like Figure 1 to Figure 3 As shown, during the processing of disparity data, the neural network processing unit NPU can first obtain multiple real-scene images with parallax from the camera module, and extract the first feature map and the second feature map indicating the parallax of the original image through a pre-trained deep neural network (e.g., BackBone).
[0039] Taking the binocular camera system of AR / VR / MR and other devices as an example, the multiple original images may include a left-eye image captured by a left-eye camera and a right-eye image captured by a right-eye camera. Correspondingly, the first feature map and the second feature map may be a left-eye feature map and a right-eye feature map indicating the binocular disparity of the binocular camera system, respectively. Here, the left-eye feature map and the right-eye feature map may be represented by a multi-channel tensor in the format of NCHW, where N is the stride feature dimension, C is the channel feature dimension, H is the height feature dimension, and W is the width feature dimension. The height feature dimension H and the width feature dimension W are generally smaller than the resolution of the original image. The amount of data of the right-eye feature map can be configured as an integer multiple of the amount of data of the left-eye feature map according to the stride (Stride) of the disparity processing. For example, in an embodiment where N=4, the amount of data of the right-eye feature map can be correspondingly configured as 4 times the amount of data of the left-eye feature map. In view of the fact that the specific steps of extracting the first feature map and the second feature map indicating the disparity of the original image via the neural network processing unit NPU are prior art in the art, they will not be elaborated here.
[0040] Afterwards, the neural network processing unit NPU can store the extracted first feature map and the second feature map into the dynamic memory DDR for extraction when the digital signal processor DSP heterogeneously calculates the cost volume features.
[0041] Please refer to Figure 4 , Figure 4 A data interaction diagram of a method for processing disparity data provided by some embodiments of the present invention is shown. Figure 4 In the illustrated embodiment, the dynamic memory DDR may be further configured with a plurality of storage subunits 411, 412. The neural network processing unit NPU may store the extracted first feature map into the storage subunit 411 of the dynamic memory DDR, and store the extracted second feature map into the storage subunit 412 of the dynamic memory DDR.
[0042] Furthermore, the static memory SRAM may also be configured with a plurality of storage subunits 421, 422. Before the heterogeneous computation of the cost space feature, the static memory SRAM may execute step S1, and load the first feature map and the second feature map stored in each storage subunit 411, 412 of the dynamic memory DDR to the corresponding storage subunits 421, 422 in the static memory SRAM in advance by means of data pre-fetching, so as to reduce the bandwidth load of the dynamic memory DDR, thereby further improving the efficiency and real-time performance of data processing.
[0043] After that, if Figures 1 to 4As shown, in the process of heterogeneously calculating cost space features, the digital signal processor DSP can preferably execute step S2, obtain the first feature map and the second feature map stored in each storage subunit 421, 422 of the static memory SRAM, and then slice the first feature map and the second feature map according to at least one dimension of the first feature map and the second feature map, and store the obtained multiple groups of slice feature data Tile Left and Tile Right in multiple first memory subunits of the first memory TCM0 in the digital signal processor DSP respectively.
[0044] Specifically, in some embodiments, the first feature map and the second feature map respectively include four-dimensional feature data of stride feature dimension N, channel feature dimension C, height feature dimension H, and width feature dimension W. Here, for an embodiment in which the original image height (H)*width (W) is 640*480, the disparity (D) is 320, and the stride (Stride) is 4, the feature dimension of the first feature map is 1*16*120*160, and the feature dimension of the second feature map is 1*16*120*640.
[0045] The digital signal processor DSP can first obtain the disparity feature data of the disparity feature dimension D, and slice (Tile) the first feature map according to the disparity feature dimension D and the channel feature dimension C, divide the first feature map into multiple groups of slice feature data, and then store one group of the first slice feature data into the Tile Left memory subunit of the first memory TCM0.
[0046] In addition, the digital signal processor DSP can also perform slicing (Tile) processing on the second feature map according to the disparity feature dimension D and the channel feature dimension C, divide the second feature map into multiple groups of slice feature data, and store one group of second slice feature data into the Tile Right memory subunit of the first memory TCM0. Here, for the above embodiment with a stride of 4, the size of the Tile Right memory subunit is 4 times that of the Tile Left memory subunit.
[0047] Afterwards, the digital signal processor DSP can calculate the absolute values of the differences between multiple groups of first slice feature data Tile Left of the first feature map and multiple groups of second slice feature data Tile Right of the corresponding dimensions of the second feature map based on the storage data in multiple first memory sub-units of the first memory TCM0, and store the absolute values in multiple second memory sub-units of the second memory TCM1, and then perform cumulative calculation on the storage data in the multiple second memory sub-units based on at least one dimension to determine the cost space (Cost Volume) feature indicating the parallax of the original image.
[0048] Specifically, if Figure 4 As shown, the digital signal processor DSP can first prepare four second memory sub-units Acc0~Acc3 in the second memory TCM1, and interleave the second feature map according to the width feature dimension W to convert it into four intermediate feature maps R Shuffle0~R Shuffle3 of the same size as the first feature map.
[0049] Afterwards, the digital signal processor DSP may adopt the SUB operator to respectively subtract each intermediate feature map from the first feature map to obtain corresponding differences, and then store the differences in each corresponding second memory sub-unit Acc0 to Acc3.
[0050] Furthermore, in order to avoid the introduced disparity feature data from raising the cost space operation to a five-dimensional feature operation, the digital signal processor DSP can also pre-slice the second feature map, and use the second NCHW tensor to perform a segmentation process on the second slice feature data based on a preset stride and a width feature dimension W, so as to obtain multiple groups of second segmentation feature data of the second slice feature data under the preset stride:
[0051] right_slice_0=right[:,:,:,0:W:4]
[0052] right_slice_1=right[:,:,:,1:W:4]
[0053] right_slice_2=right[:,:,:,2:W:4]
[0054] right_slice_3=right[:,:,:,3:W:4].
[0055] In addition, the digital signal processor DSP can also perform a segmentation process on the first feature map, and use the first NCHW tensor to perform a segmentation process on the first slice feature data based on the width feature dimension W to obtain multiple groups of first segmentation feature data:
[0056] left_slice=left[:,:,:,i:].
[0057] In this way, the digital signal processor DSP can calculate the difference between each group of first segmentation feature data and the second segmentation feature data of the corresponding width feature dimension with respect to the channel feature dimension C based on the first NCHW tensor and the second NCHW tensor.
[0058] After that, the digital signal processor DSP can execute step S3, use the ABS operator to perform absolute value operation on the difference data stored in each second memory sub-unit Acc0~Acc3, and then execute step S4, use the ADD operator to perform accumulation operation on the channel feature dimension C on the absolute value data stored in each second memory sub-unit Acc0~Acc3 to determine the accumulated value of each absolute value, that is,
[0059] volume[:,i*4+0,:,:]=Sum(Abs(left_slice-right_slice_0[:,:,:,0:Wi])
[0060] volume[:,i*4+1,:,:]=Sum(Abs(left_slice-right_slice_1[:,:,:,0:Wi])
[0061] volume[:,i*4+2,:,:]=Sum(Abs(left_slice-right_slice_2[:,:,:,0:Wi])
[0062] volume[:,i*4+3,:,:]=Sum(Abs(left_slice-right_slice_3[:,:,:,0:Wi])
[0063] Among them, Sum represents the accumulation operation on the channel feature dimension C, Abs represents the absolute value operation, and the operation result indicates a cost space feature of the original image disparity along the corresponding channel feature dimension C.
[0064] Afterwards, the digital signal processor DSP can determine the number of inner loops (i.e., Loop for Channel = 16) according to the number of channels (C = 16), and determine the number of outer loops (i.e., Loop for Disparity = 320 / 4 = 80) according to the preset disparity range (i.e., D = 320) and the preset stride (i.e., Stride = 4), and then perform accumulation operations in sequence according to the number of inner loops and the number of outer loops to determine the cost space features of the above-mentioned disparity feature data.
[0065] Further, in Figure 4In the illustrated embodiment, the dynamic memory DDR may also be preferably configured with a plurality of storage subunits 413, and the static memory SRAM may also be preferably configured with a plurality of storage subunits 423. In the process of calculating the above cost space features, the digital signal processor DSP may also preferably combine the storage subunits 413 and / or 423 configured in the dynamic memory DDR and / or the static memory SRAM to further reduce the write-back process of the processor DSP and the dynamic memory DDR, so as to further improve the efficiency and real-time performance of data processing.
[0066] Specifically, in the process of determining the cost space characteristics, the digital signal processor DSP can first use the ADD operator to perform cumulative calculations on the storage data in multiple second memory sub-units Acc0~Acc3 based on the channel feature dimension C, so as to respectively determine the inner loop cumulative values of each absolute value with respect to the channel feature dimension C.
[0067] After completing the inner loop accumulation operation of a channel feature dimension C, the digital signal processor DSP can store the calculated inner loop accumulation value in multiple storage sub-units 423 of the static memory SRAM, and initialize each second memory sub-unit Acc0~Acc3 to perform the inner loop accumulation calculation based on the next channel feature dimension C.
[0068] Then, the digital signal processor DSP can also perform outer loop accumulation calculation on the storage data in each storage subunit 423 based on the disparity feature dimension D, and determine the outer loop accumulation value about the disparity feature dimension D according to the outer loop accumulation operation result of each channel feature dimension C in a disparity feature dimension D.
[0069] After completing the outer loop accumulation operation of a disparity feature dimension D, the digital signal processor DSP can execute step S5, store the outer loop accumulation value into multiple storage sub-units 413 of the dynamic memory DDR, and initialize each storage sub-unit 423 as described above to perform an outer loop accumulation calculation based on the next disparity feature dimension D until the outer loop accumulation operations of all disparity feature dimensions D are completed.
[0070] In this way, after completing the outer loop accumulation operation of each disparity feature dimension D, the digital signal processor DSP can determine the cost space features indicating the disparity of the original image based on the outer loop accumulation values stored in each storage sub-unit 413 of the dynamic memory DDR, thereby making full use of the on-chip storage resources of the digital signal processor DSP and the bandwidth of the dynamic memory DDR to further improve the overall data processing efficiency and real-time performance of the disparity data processing system.
[0071] Furthermore, in some embodiments, the digital signal processor DSP can preferably write back and store the accumulated values of each outer loop in the multiple storage subunits 413 of the dynamic memory DDR while performing the next operation. In this way, the present invention can cover the time consumption of writing back this part of data in the next operation, thereby making full use of the bandwidth of the dynamic memory DDR, so as to further improve the overall data processing efficiency and real-time performance of the parallax data processing system.
[0072] In addition, in some embodiments, the above-mentioned difference operation, accumulation operation and absolute value operation can preferably be implemented by at least one hardened computing circuit such as a subtraction operation circuit, an absolute value operation circuit, an addition operation circuit, an accumulator circuit, etc. configured in the digital signal processor DSP, thereby further reducing the software processing load of the digital signal processor DSP.
[0073] Therefore, compared with the prior art which first adds disparity dimension (Disparity) data to the four-dimensional tensor (Tensor) of NCHW, and then calculates the cost space based on the five-dimensional NCDHW tensor and memory access operators such as Gather and Scatter, the present invention, on the one hand, exports the calculation process of the cost space to the digital signal processor DSP for heterogeneous calculations to reduce the hardware design complexity of the NPU, and on the other hand, merges the D-dimensional features into the C-dimensional features, and then implements the calculation function of the cost space through basic operators such as ABS, ADD, SUB, etc., to overcome the dependence of existing deep learning algorithms on memory access operators, and directly outputs four-dimensional cost space features, thereby greatly reducing the data calculation load and the output data volume in the cost space calculation process.
[0074] In addition, if Figure 1 and Figure 3 As shown, after completing the accumulation operation of the outer loop, the neural network processing unit NPU can also obtain the stored outer loop accumulation value through the multiple storage subunits 413 of the dynamic memory DDR to determine the NCHW four-dimensional cost space feature indicating the disparity of the original image, and perform cost aggregation on the four-dimensional cost space feature to calculate the disparity (Disparity) of the above-mentioned multiple real-scene images. In view of the specific scheme of cost aggregation of cost space features through the neural network processing unit NPU to calculate the disparity of multiple real-scene images is a prior art in the field, it will not be elaborated here.
[0075] In summary, the above-mentioned disparity data processor, disparity data processing system, disparity data processing method, and computer-readable storage medium provided by the present invention can reduce the hardware design complexity of NPU, reduce the amount of output data, and reduce the bandwidth pressure of external storage by heterogeneously calculating the cost space features in the digital signal processor (DSP). In addition, by using basic operators such as ABS, ADD, SUB to perform cost space operations, the present invention can also overcome the dependence of deep learning algorithms on memory access operators such as Gather or Scatter, thereby improving the MAC utilization of the neural network processing unit NPU. In addition, by configuring the first memory TCM0 and the second memory TCM1 on the digital signal processor DSP, configuring multiple storage subunits 411~413 on the dynamic memory DDR, configuring multiple storage subunits 421~423 on the static memory SRAM, and configuring pre-fetch (Pre-fetch) and layer-by-layer cache accumulation data transmission methods, the present invention can also make full use of the on-chip storage resources of the digital signal processor DSP to reduce the write-back process of external storage, thereby improving the efficiency and real-time performance of data processing.
[0076] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.
[0077] Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips cited throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.
[0078] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.
[0079] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.
[0080] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.
[0081] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A disparity data processor, characterized in that: The processor is configured with a first memory and a second memory, and is configured as follows: Acquire a first feature map and a second feature map indicating the parallax of the original image, wherein the data volume of the second feature map is configured to be an integer multiple of the data volume of the first feature map according to the stride of the parallax processing; According to at least one dimension of the first feature map and the second feature map, slicing the first feature map and the second feature map, and storing the obtained multiple groups of slice feature data in multiple first memory sub-units of the first memory respectively; Based on the stored data in the plurality of first memory subunits, respectively calculating absolute values of differences between the plurality of groups of first slice feature data of the first feature graph and the plurality of groups of second slice feature data of corresponding dimensions of the second feature graph, and storing the absolute values in the plurality of second memory subunits of the second memory; and Based on the at least one dimension, the stored data in the plurality of second memory sub-units are cumulatively calculated, and a cost space feature indicating the parallax of the original image is determined according to the cumulative value of each of the absolute values.
2. The processor according to claim 1, wherein: The processor is connected to an external static memory, and the step of obtaining a first feature map and a second feature map indicating the parallax of the original image includes: The first characteristic map and the second characteristic map stored in the static memory are obtained through the static memory.
3. The processor according to claim 2, wherein: The first feature map and the second feature map respectively include a stride feature dimension N, a channel feature dimension C, a height feature dimension H, and a width feature dimension W. The step of slicing the first feature map and the second feature map according to at least one dimension of the first feature map and the second feature map includes: Obtaining disparity feature data of disparity feature dimension D; and According to the disparity feature dimension D and the channel feature dimension C, the first feature map and the second feature map are respectively divided into a plurality of groups of slice feature data.
4. The processor according to claim 3, wherein: The step of respectively calculating the absolute values of the differences between the plurality of groups of first slice feature data of the first feature graph and the plurality of groups of second slice feature data of the corresponding dimensions of the second feature graph based on the stored data in the plurality of first memory sub-units comprises: Using a first NCHW tensor, performing segmentation processing based on the width feature dimension W on the first slice feature data to obtain multiple groups of first segmentation feature data; Using a second NCHW tensor, segmenting the second slice feature data based on a preset stride and the width feature dimension W to obtain a plurality of groups of second segmentation feature data of the second slice feature data at the preset stride; and Based on the first NCHW tensor and the second NCHW tensor, the absolute value of the difference between each group of the first segmentation feature data and the second segmentation feature data of the corresponding width feature dimension with respect to the channel feature dimension C is calculated respectively.
5. The processor according to claim 4, wherein: The step of performing cumulative calculation on the stored data in the plurality of second memory sub-units based on the at least one dimension, and determining the cost space feature indicating the parallax of the original image according to the cumulative value of each of the absolute values comprises: Determining the number of cycles according to a preset parallax range and the preset stride; According to the number of loops, cyclically accumulate the absolute values to determine a plurality of third NCHW tensors, wherein each of the third NCHW tensors indicates a cost space feature of the original image disparity along a corresponding channel feature dimension C; and A fourth NCHW tensor is calculated according to the plurality of third NCHW tensors to serve as the cost space feature.
6. The processor according to claim 3, wherein: The step of performing cumulative calculation on the stored data in the plurality of second memory sub-units based on the at least one dimension, and determining the cost space feature indicating the parallax of the original image according to the cumulative value of each of the absolute values comprises: Based on the channel feature dimension C, performing cumulative calculation on the stored data in the plurality of second memory sub-units to respectively determine inner loop cumulative values of the absolute values with respect to the channel feature dimension C; Storing each of the inner loop accumulated values into a plurality of third storage subunits of the static memory; Initializing the plurality of second memory sub-units to perform inner loop accumulation calculation based on the next channel feature dimension C; Based on the parallax feature dimension D, performing cumulative calculation on the stored data in the plurality of third storage sub-units to respectively determine outer-layer loop cumulative values of each of the inner-layer loop cumulative values with respect to the parallax feature dimension D; and According to each of the outer layer loop accumulated values, a cost space feature indicating the original image disparity is determined.
7. The processor according to claim 6, wherein: The processor is also connected to an external dynamic memory, wherein the static memory obtains the first feature map and the second feature map from the dynamic memory by way of data pre-fetching, and the processor is further configured as: While the next operation is being performed, each of the outer loop accumulated values is written back and stored in a plurality of fourth storage sub-units of the dynamic memory, so that the operation time of the next operation is used to cover the write-back time and the storage write-back time.
8. The processor according to claim 7, wherein: The dynamic memory is further configured with a plurality of fifth storage sub-units, and the processor is further connected to an external neural network processing unit via the dynamic memory, wherein the neural network processing unit is configured as follows: Acquire multiple real-scene images, and perform feature extraction on them to determine the first feature map and the second feature map; storing the first characteristic map and the second characteristic map in the plurality of fifth storage sub-units respectively for being called by the static memory; Obtaining, via the plurality of fourth storage subunits of the dynamic memory, the outer loop accumulated values stored therein, to determine a cost space feature indicating the disparity of the original image; and Cost aggregation is performed on the cost space features to calculate the disparity of the multiple real scene images.
9. The processor according to any one of claims 1 to 8, wherein: The processor is further configured with at least one hardened calculation circuit, wherein the hardened calculation circuit is selected from at least one of a subtraction operation circuit, an absolute value operation circuit, an addition operation circuit, and an accumulator circuit.
10. A disparity data processing system, characterized in that: include: A neural network processing unit, configured to perform feature extraction on the acquired multiple real scene images to determine a corresponding first feature map and a second feature map, and perform cost aggregation on cost space features determined based on the first feature map and the second feature map to calculate the disparity of the multiple real scene images; as well as The disparity data processor according to any one of claims 1 to 9, configured to determine corresponding cost space features according to the first feature map and the second feature map.
11. The parallax data processing system according to claim 10, characterized in that: Also includes: A static memory is connected to the neural network processing unit and the processor, and is configured with multiple third storage sub-units and / or multiple sixth storage sub-units, wherein the multiple third storage sub-units are used to store inner loop cumulative values determined by accumulating the storage data in the multiple second memory sub-units of the processor, and the multiple sixth storage sub-units are used to pre-access the first feature map and the second feature map.
12. The parallax data processing system according to claim 11, characterized in that: Also includes: A dynamic memory is connected to the neural network processing unit and the processor, and is configured with multiple fourth storage sub-units and / or multiple fifth storage sub-units, wherein the multiple fourth storage sub-units are used to store outer loop cumulative values determined by summing up the storage data in the multiple third storage sub-units, and the multiple fifth storage sub-units are used to store the first feature map and the second feature map extracted by the neural network processing unit.
13. A method for processing disparity data, characterized in that: The following steps are involved: Acquire a first feature map and a second feature map indicating the parallax of the original image, wherein the data volume of the second feature map is configured to be an integer multiple of the data volume of the first feature map according to the stride of the parallax processing; According to at least one dimension of the first feature map and the second feature map, the first feature map and the second feature map are sliced, and the obtained multiple groups of slice feature data are respectively stored in multiple first memory subunits of the first memory of the processor; Based on the stored data in the plurality of first memory sub-units, respectively calculating absolute values of differences between the plurality of groups of first slice feature data of the first feature graph and the plurality of groups of second slice feature data of corresponding dimensions of the second feature graph, and storing the absolute values in the plurality of second memory sub-units of the second memory of the processor; and Based on the at least one dimension, the stored data in the plurality of second memory sub-units are cumulatively calculated, and a cost space feature indicating the parallax of the original image is determined according to the cumulative value of each of the absolute values.
14. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method for processing disparity data as claimed in claim 13 is implemented.
Citation Information
Patent Citations
Image three-dimensional representation method based on parallax hierarchical segmentation, and system
CN104820981A
Parallax image acquisition method, device and apparatus and control system
CN109919993A
Binocular matching method and device, equipment and storage medium
CN113393366A
Improving the efficiency of image difference calculations
CN113785323A
Joint depth prediction from dual camera and dual pixel
CN114556422A
