Configurable real-time parallax point cloud computing device and method

Abstract

This invention provides a configurable real-time parallax point cloud computing device and method, including an image caching unit, a caching controller, a PE array, a result shaping module, a minimum value search module, and a configuration parsing module. The image caching unit is used to output image window data with a specified window size and sliding window order. The caching controller is used to control the image caching unit to output image window data and distribute it to the PEs in the PE array. The PE array is used to generate several PUs with a specified structure and obtain the SAD matching cost calculation result. The result shaping module is used to add data fields to the matching cost. The minimum value search module is used to search for the minimum value of the matching cost step by step to obtain the disparity value. The configuration parsing module is used to parse the received configuration information, generate corresponding control signals, and input them to other modules. This enables real-time parallax point cloud computing, with configurable matching parameters that do not require reconstruction.

Description

Technical Field

[0001] This invention relates to the field of microelectronics technology, and in particular to a configurable real-time parallax point cloud computing device and method. Background Technology

[0002] Stereo matching is a crucial step in binocular stereo vision. Stereo matching algorithms search for corresponding points in the left and right images based on pixel similarity to determine disparity. By searching for corresponding points across all pixels in the entire image, a disparity point cloud can be generated for the entire image, which can then be used for tasks such as ranging or 3D reconstruction. Stereo matching algorithms can be deployed on various platforms, including Central Processing Units (CPUs), Graphics Processing Units (GPUs), Field-Programmable Gate Arrays (FPGAs), and Application-Specific Integrated Circuits (ASICs).

[0003] CPUs and GPUs offer good programmability, allowing them to adapt to different matching parameters and meet the needs of various 3D matching tasks. However, their real-time performance is poor, failing to meet the demands of high-real-time applications. ASICs offer high energy efficiency and real-time performance, but their flexibility is limited, making them unable to adapt to different matching parameters. FPGAs can effectively accelerate computationally intensive tasks, but current technologies can only adapt to different matching parameters through reconfiguration, resulting in significant time costs associated with reconfiguration designs. Summary of the Invention

[0004] To address the problems existing in the prior art, the present invention provides a configurable real-time parallax point cloud computing device and method.

[0005] In a first aspect, the present invention provides a configurable real-time parallax point cloud computing device, comprising:

[0006] Image caching unit, cache controller, processing unit PE array, result shaping module, minimum value search module, and configuration parsing module;

[0007] The image caching unit is connected to the caching controller and is used to shape the cached binocular image data according to a specified window size and sliding window order under the control of the caching controller, and then output the image window data to the caching controller.

[0008] The cache controller is connected to the configuration parsing module and the PE array respectively, and is used to control the image cache unit to output image window data according to the control signal transmitted by the configuration parsing module. The image window data is distributed to the PEs in the PE array by the cache controller.

[0009] The PE array is connected to the configuration parsing module and the result shaping module respectively, and is used to generate a number of PUs with a specified structure according to the control signal transmitted by the configuration parsing module, and process the input image window data based on the number of PUs with the specified structure to obtain the SAD matching cost calculation result and output it to the result shaping module.

[0010] The result shaping module is connected to the configuration parsing module and the minimum value search module respectively, and is used to add fields to the input SAD matching cost calculation result according to the control signal transmitted by the configuration parsing module and then output it to the minimum value search module.

[0011] The minimum value search module is connected to the configuration parsing module and is used to search for the minimum value step by step based on the control signal and minimum value search algorithm transmitted by the configuration parsing module, and output the disparity value corresponding to the minimum matching cost.

[0012] The configuration parsing module is used to parse the received configuration information and generate corresponding control signals, which are then input to the cache controller, the PE array, the result shaping module, and the minimum value search module.

[0013] Optionally, the PEs in the PE array are interconnected vertically and horizontally, with intermediate results being passed in the vertical direction and operands and matching costs being passed in the horizontal direction.

[0014] Optionally, the PE array includes one or more of the following types of PEs:

[0015] Ultra PE is used to perform the absolute difference of two operands and the accumulation of partial sums during the SAD matching cost calculation process.

[0016] Standard PE is used to perform the subtraction and absolute value calculation of two operands, as well as the accumulation operation of partial sums, during the SAD matching cost calculation process.

[0017] Lite PE is used to perform the operation of taking the absolute value of the difference between two operands during the SAD matching cost calculation process;

[0018] The computing resources corresponding to the Ultra PE are greater than those of the Standard PE.

[0019] Optionally, each column in the PE array can be configured as one or more PUs, which are used to perform SAD matching cost calculation operations with a specified window size.

[0020] Optionally, in the PU, the PE in the first row is either the Ultra PE or the Standard PE.

[0021] Secondly, the present invention also provides a configurable real-time parallax point cloud computing method, comprising:

[0022] The configuration parsing module parses the received configuration information and generates corresponding control signals, which are then input to the cache controller, the PE array, the result shaping module, and the minimum value search module, respectively.

[0023] The cache controller, based on the control signals transmitted by the configuration parsing module, controls the image cache unit to output image window data corresponding to one or more channels of binocular image data according to the specified window size and sliding window order;

[0024] The PE array generates several PUs with a specified structure according to the control signals transmitted by the configuration parsing module, and processes the input image window data based on the several PUs with the specified structure to obtain the SAD matching cost calculation result corresponding to the image window data.

[0025] The result shaping module adds fields to the SAD matching cost calculation result output by the PE array according to the control signal transmitted by the configuration parsing module;

[0026] The minimum value search module searches for the minimum value step by step in the SAD matching cost calculation result after adding fields, based on the control signals and minimum value search algorithm transmitted by the configuration parsing module, and outputs the disparity value corresponding to the minimum matching cost.

[0027] Optionally, the configuration information includes:

[0028] Image resolution, matching window size, parallax search depth, number of channels for binocular image data, and PE working mode.

[0029] Optionally, the method for determining the configuration information includes:

[0030] Determine whether the performance metrics for single data stream or multiple data stream are met;

[0031] Computational resources are allocated based on the number of allocable unit computing resources in the PE array, the disparity search depth corresponding to each data stream, and the video frame rate corresponding to each data stream.

[0032] Configuration information is generated based on the allocated computing resources.

[0033] Optionally, when there are two data streams, the allocation of computing resources based on the number of allocable unit computing resources in the PE array, the disparity search depth corresponding to each data stream, and the video frame rate corresponding to each data stream includes:

[0034] After allocating one unit of computing resource to each data stream, the remaining allocable computing resources in the PE array are determined;

[0035] If the remaining allocable computing resources can provide at least one unit of computing resources for each data stream and the first condition is met, then continue to allocate one unit of computing resources for each data stream.

[0036] If the remaining allocable computing resources can provide at least one unit of computing resources for each data stream, but the first condition is not met, then according to the relationship between the first and second values, a unit of computing resources is allocated to each data stream individually.

[0037] The first value is determined based on the parallax search depth and video frame rate corresponding to each data stream, the second value is determined based on the number of unit computing resources currently allocated to each data stream, and the first condition is determined based on the first value, the second value, and a preset threshold.

[0038] Optionally, the method further includes:

[0039] If the remaining allocable computing resources can only provide one unit of computing resources for the target data stream, then all the remaining allocable computing resources are allocated to the target data stream.

[0040] The configurable real-time parallax point cloud computing device and method provided by this invention can adapt to different matching parameters by configuring the parsing module, without reconstructing the FPGA; the calculation of SAD matching cost is completed by the parallel pipeline structure of the PE array, which meets the high real-time requirements and ensures both adaptability to different matching information and high real-time performance. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is an overall block diagram of the configurable real-time parallax point cloud generation system based on FPGA provided by the present invention.

[0043] Figure 2 This is a schematic diagram of the configurable real-time parallax point cloud computing device provided by the present invention;

[0044] Figure 3 This is a schematic diagram of the resolution compatibility method of the image caching unit provided by the present invention;

[0045] Figure 4 This is a schematic diagram illustrating the definitions of PSAD and PSUM provided by the present invention;

[0046] Figure 5 This is a schematic diagram of the SAD matching cost calculation pipeline design provided by the present invention;

[0047] Figure 6 This is a schematic diagram of the fully parallel pipelined SAD computing array provided by the present invention;

[0048] Figure 7 This is a diagram of the internal structure of the Ultra PE provided by this invention;

[0049] Figure 8 This is a flowchart illustrating the configurable real-time parallax point cloud computing method provided by the present invention.

[0050] Figure 9 This is a schematic diagram of the resource-aware configuration generation process provided by the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] Binocular stereo vision, a common method for acquiring scene depth information, boasts good reliability and robustness, and is widely used in mobile robots, autonomous driving, industrial automation, and automatic monitoring. Stereo matching is a crucial step in binocular stereo vision. Stereo matching algorithms search for corresponding points in the left and right images based on pixel similarity to determine disparity. By searching for corresponding points across all pixels in the entire image, a disparity point cloud can be generated for the entire image, which can then be used for tasks such as ranging or 3D reconstruction. Currently, stereo matching algorithms can be categorized into local matching algorithms, global matching algorithms, and semi-global matching algorithms. Local matching algorithms, due to their unique real-time characteristics, are widely used in high real-time applications.

[0053] Stereo matching algorithms can be deployed on various platforms such as CPUs, GPUs, FPGAs, and ASICs. CPUs and GPUs offer good programmability, allowing them to adapt to different matching parameters (such as matching window size, disparity search depth, and image resolution) to meet the needs of stereo matching tasks in different scenarios. However, their real-time performance is poor, failing to meet the demands of high-real-time applications. ASICs offer high energy efficiency and real-time performance, but their flexibility is limited, making them unable to adapt to different matching parameters. FPGAs effectively accelerate computationally intensive tasks and can be reconfigured to adapt to different matching parameters, achieving a trade-off between real-time performance and flexibility. They have become the mainstream solution for accelerating stereo matching.

[0054] Currently, there are numerous FPGA-based stereo matching acceleration platforms that implement local matching algorithms such as the Sum of Absolute Differences (SAD), Sum of Squared Differences (SSD), and Census Transform. However, these platforms can only adapt to different matching parameters by redesigning the design, and the time cost of redesigning the design is relatively large. Therefore, this invention provides an FPGA-based stereo matching algorithm solution that can simultaneously meet the requirements of real-time performance, flexible adaptation to different matching parameters, and no need for redesign in stereo matching tasks.

[0055] The core idea of ​​this invention is to adapt to different matching parameters by configuring the parsing module without refactoring the FPGA; and to complete the SAD matching cost calculation through the PE array pipeline structure to ensure high real-time performance.

[0056] Figure 1 This is an overall block diagram of the configurable real-time parallax point cloud generation system based on FPGA provided by the present invention. Figure 1As can be seen, image data acquired by the left and right camera lenses and the acquisition chip is transmitted to the FPGA chip via high-speed interfaces, such as Mobile Industry Processor Interface (MIPI), Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), and DisplayPort (DP). After passing through the basic image processing module, it enters the image distortion correction module for imaging distortion correction. The output corrected pixel data is cached in frames to external memory in a ping-pong buffer under the scheduling of the cache controller. The number of images cached in the external memory is sent to the configurable disparity point cloud computing module for stereo matching calculation. Stereo matching uses a local matching algorithm, with SAD (Self-Aparing Disparity) as the metric to measure the matching cost. The obtained disparity data is output through a high-speed interface (such as Peripheral Component Interconnect Express, PCIe).

[0057] The main control unit (e.g., a CPU built into an FPGA) generates configuration information and sends it to each configurable module to configure matching parameters. The camera and acquisition chip can adjust different resolutions and frame rates through register configuration. The basic image processing module includes demos, grayscale correction, image color format conversion, and resolution cropping. Cropping from high resolution to low resolution can be achieved through register configuration. The image distortion correction module can achieve compatibility with different resolutions through register configuration.

[0058] The configurable real-time parallax point cloud generation system based on FPGA provided by this invention can configure different resolutions, matching window widths, and parallax search depths by configuring registers, and can support real-time processing of multiple sets of binocular data.

[0059] Figure 2 This is a schematic diagram of the structure of the configurable real-time parallax point cloud computing device provided by the present invention. Figure 2 As can be seen from the diagram, the device can be applied to a configurable real-time parallax point cloud generation system based on FPGA. The device includes an image caching unit 200, a caching controller 210, a processing element (PE) array 220, a result shaping module 230, a minimum value search module 240, and a configuration parsing module 250.

[0060] The image buffer unit 200 is connected to the buffer controller 210 and is used to shape the buffered binocular image data according to the specified window size and sliding window order under the control of the buffer controller 210 and then output the image window data to the buffer controller 210.

[0061] The cache controller 210 is connected to the configuration parsing module 250 and the PE array 220 respectively, and is used to control the image cache unit 200 to output image window data according to the control signal transmitted by the configuration parsing module 250. The image window data is distributed to the PE in the PE array 220 via the cache controller 210.

[0062] The PE array 220 is connected to the configuration parsing module 250 and the result shaping module 230 respectively. It is used to generate a number of algorithm processing units (PUs) with a specified structure according to the control signals transmitted by the configuration parsing module 250, and process the input image window data based on the number of PUs with the specified structure to obtain the SAD matching cost calculation result and output it to the result shaping module 230.

[0063] The PE array adopts a mesh topology structure, with each PE interconnected with its four PEs above, below, left, and right. The PU with a specified structure refers to an algorithm processing unit containing a column and several rows of PEs, used to complete the SAD matching cost calculation for a specified window and obtain the calculation result. The number of rows can be determined according to the image window size in the configuration information. For example, if the image window size is 3×3, then a PU contains 1 column and 3 rows of PEs.

[0064] SAD matching cost calculation refers to the sum of the absolute values ​​of the differences between corresponding pixel values ​​in two windows (e.g., left and right eye image windows).

[0065] The result shaping module 230 is connected to the configuration parsing module 250 and the minimum value search module 240 respectively. It is used to add data fields to the input SAD matching cost calculation result according to the control signal transmitted by the configuration parsing module 250 and then output it to the minimum value search module 240.

[0066] The minimum value search module 240 is connected to the configuration parsing module 250 and is used to search for the minimum value step by step based on the minimum value search tree, and output the disparity value corresponding to the minimum matching cost.

[0067] The configuration parsing module 250 is used to parse the received configuration information and generate corresponding control signals, which are then input to the cache controller, PE array, result shaping module, and minimum search module.

[0068] from Figure 2As can be seen, the image cache unit 200 has a storage size of 48.64KB, including 38 banks. Each bank consists of a 1280-byte dual-port block random access memory (BRAM), capable of storing 1280 bytes of pixel data. For example, it can store one row of pixel data for each of the left and right images at a 480p resolution (i.e., 640×480), half a row of pixel data for each of the left and right images at a 720p resolution (i.e., 1280×720), or one-third a row of pixel data for each of the left and right images at a 1080p resolution (i.e., 1920×1080). However, those skilled in the art should understand that the storage size, number of banks, and bank size of the image cache unit 200 are not limiting and can be flexibly adjusted as needed.

[0069] Figure 3 This is a schematic diagram of the resolution compatibility method of the image caching unit provided by the present invention. As can be seen from the figure, the image caching unit 200 can perform shaping operations on images of different resolutions, that is, retrieve pixel data or rearrange pixels according to a fixed window size and sliding window order. After shaping, the matching window size is 2n+1 (n∈N). * (Note: At a resolution of 1920×1080, the size of the matching window after reshaping is 2n+1 (n∈N)) * (n≤6). For a single image, each row of the shaping buffer can cache 640 pixels, which can complete the shaping operation for an image with 640*n (n is a positive integer) rows of pixels. For example, the image is cached row by row according to pixel coordinates, starting from Bank0. If the input image is 480p resolution, each row of pixels occupies one Bank, and data shaping is completed by reading data from each bank simultaneously; if the input image is 720p resolution, each row of pixels occupies two Banks, and data shaping is completed by reading data from the (2n-1)th (n is a positive integer)th Bank simultaneously; if the input image is 1080p resolution, each row of pixels occupies three banks, and data shaping is completed by reading data from the (3n-1)th (n is a positive integer)th Bank simultaneously. When all Banks are full, the input data will overwrite the historical data starting from Bank0, and the row-by-row caching will continue.

[0070] The cache controller 210 can generate read and write control signals for the image cache unit based on the control signals transmitted by the configuration parsing module 250, and read and write data in the BRAM (e.g., read from external memory, write to BRAM, read from BRAM, distribute to PE array, etc.).

[0071] The PE array 220 performs SAD-based matching cost calculation. Each PE in the PE array can be configured to different operating modes according to the control signals transmitted by the configuration parsing module 250. The computational tasks of each PE differ under different operating modes (different window widths, parallax search depths, resolutions, and number of video streams), and the required FPGA resources also vary. Based on all the computational parameters supported by the architecture, a configuration space is constructed for each PE, and three types of PE structures are designed based on this configuration space: Ultra PE, Standard PE, and Lite PE. These three types of PEs are distributed in different rows of the array according to different computational tasks. More complex calculations are performed by Ultra PEs, followed by Standard PEs, and simple calculations are performed by Lite PEs, which can maximize the conservation of FPGA resources. Simultaneously, each PE in the PE array is interconnected horizontally and vertically, allowing the transmission of operands and matching costs in the horizontal direction and intermediate results in the vertical direction, thereby realizing a multi-stage pipeline design for SAD matching cost calculation and accelerating the SAD matching cost calculation. Each column in the PE array can be configured to include one or more PUs to achieve matching cost calculation at different search depths.

[0072] The result shaping module 230 adds fields to the SAD calculation results generated by the Ultra PE and Standard PE rows. The SAD matching cost calculation results generated by the Ultra PE and Standard PE rows are transmitted to the result shaping module via the horizontal interconnect data channel. The result shaping module, in conjunction with the control signals transmitted by the configuration parsing module, adds a position field to the SAD result corresponding to the candidate disparity (the difference in the horizontal coordinates of the center points of the left and right windows). That is, an 8-bit binary code is added before each result to represent the matching cost of the SAD value under the candidate disparity. The matching cost after adding the field is then sent to the minimum value search module.

[0073] The minimum value search module 240 uses a minimum value search tree to search for the minimum value in the matching cost level by level, and outputs the disparity value corresponding to the minimum matching cost.

[0074] The configuration parsing module 250 receives configuration information sent by the CPU, including the number of video streams to be processed, resolution, search depth, and matching window size. The configuration parsing module 250 parses the received configuration information, generates corresponding flag signals, and provides these signals to the cache controller 210 for generating read / write addresses and read / write enable signals; it also transmits control signals to the PE array 220, the result shaping module 230, and the minimum value search module 240.

[0075] The configuration parsing module 250 can adapt to different matching parameters without refactoring the FPGA; the SAD matching cost is calculated through the PE array pipeline structure to ensure real-time performance, thereby overcoming the shortcomings of existing technologies that cannot simultaneously achieve real-time performance and adapt to different matching information.

[0076] Optionally, the PEs in the PE array are interconnected vertically and horizontally, with intermediate results being passed in the vertical direction and operands and final matching costs being passed in the horizontal direction.

[0077] Specifically, in the calculation of the SAD matching cost, each PE in the PE array is interconnected vertically and horizontally. This allows for the transmission of operands and matching costs in the horizontal direction and intermediate results in the vertical direction. Horizontally, the data reuse feature of the three-dimensional matching pipeline mode is fully utilized to transmit operands and the final SAD matching cost within the PE array, avoiding long-distance data access to memory. Vertically, the transmission and accumulation of partial sums during the calculation of the left and right window matching costs are completed.

[0078] Figure 4 This is a schematic diagram illustrating the definitions of PSAD and PSUM provided by the present invention. From... Figure 4 As can be seen, PSAD means calculating the absolute difference of pixel values ​​at the same corresponding position in the same column of the left and right images, and then summing the absolute differences of this column.

[0079] Figure 5 This is a schematic diagram of the SAD matching cost calculation pipeline design provided by the present invention. First, the SAD calculation process is explained: For two input images (left and right), each pixel in the left image (called an anchor point) is scanned sequentially. While scanning each pixel in the left image, the following operations are performed: A matching window of a fixed size (e.g., 3×3, 5×5, etc.) is constructed centered on each anchor point, and all pixels within the window's coverage area are selected. Similarly, the window is used to cover the corresponding position in the right image, and all pixels within the coverage area are selected. The absolute value of the difference between the grayscale values ​​of the corresponding pixels in the left and right image coverage areas is calculated, and these absolute values ​​are added together. The right image coverage area is moved to the left with a step size of 1, and all pixels within the coverage area are extracted, and the SAD value is calculated. The previous step is repeated until the center position of the right image coverage area exceeds the disparity search range. The window corresponding to the minimum SAD value within this range is found; its center point is the corresponding point of the anchor point in the left image. The difference between the horizontal coordinate of the left image anchor point and its corresponding point in the right image is the disparity of that anchor point.

[0080] Figure 5Part (a) illustrates the disparity search calculation process for an anchor point with a pixel value of 98 (the center point of a 3×3 window) in the left image. The window size is 3×3, and the search range is 4. During the search, the SAD values ​​of the left image window data and right image windows 1, 2, 3, and 4 are calculated sequentially, and the minimum SAD value is obtained to determine the disparity. Figure 5 Part (b) illustrates the implementation process of the SAD computation pipeline in the computing architecture proposed in this invention. For example, the SAD parallel process is as follows: the SAD computation process performed by two 3×3 windows can be divided into three sub-processes, which are named PSAD processes. The result of the PSAD process is defined as PSUM, and the SAD computation result can be obtained by adding several PSUMs together.

[0081] Figure 5 The calculation process shown in part (a) uses a complete SAD calculation as its basic granularity, which results in repeated data accesses. To achieve efficient pipelining, the SAD calculation process is now adjusted to... Figure 5 The form shown in part (b) uses a single PSAD calculation as its granularity, and implements the SAD calculation process through the accumulation of PSUM. Specifically, at time t1, the sub-window shown in ① in the left figure and the four sub-windows shown in ① in the right figure execute four PSAD processes in parallel, producing four PSUMs, which are PSUM1_1, PSUM1_2, PSUM1_3, and PSUM1_4 from right to left (PSUMn_m: the PSAD result of the m-th sub-window from right to left in the set of sub-windows shown in the left figure n and the right figure n); at time t2, the sub-window shown in ② in the left figure and the four sub-windows shown in ② in the right figure execute four PSAD processes in parallel, producing four PSUMs, which are PSUM2_1, PSUM2_2, PSUM2_3, PSUM2_4, and so on from right to left. Under this calculation method, Figure 5 The SAD process for window 1 shown in part (a) can be obtained by adding PSUM1_1, PSUM2_1, and PSUM3_1, denoted as SUM. This pipelined approach can fully utilize the data reusability of the right figure. When calculating the matching cost of two windows with a disparity of 0 centered at 90° in the left figure, SUM is added to PSUM4_1 (… Figure 5 In part (b), the PSAD result of the first sub-window from right to left in the set of sub-windows shown in ④ of the left figure and ④ of the right figure is subtracted from PSUM1_1 to avoid redundant calculations. This calculation method requires local caching of historical PSUM.

[0082] For example, Figure 6This is a schematic diagram of the fully parallel pipelined SAD computing array provided by the present invention. The array consists of three rows and four columns of PEs and can be configured as four PUs (PU1, PU2, PU3 and PU4) to complete a fully parallax parallel computing process with a window size of 3×3 and a search range of four. Figure 5 In part (b), each sub-window in the left image needs to perform PSAD calculations with the four sub-windows in the right image. Each PSAD calculation is completed by one column of PEs (i.e., one PU). The number of PEs in each column is determined by the window size. Therefore, four columns of such PEs are needed to complete this full parallax parallel calculation. Data can be transferred between PEs in the up, down, left, and right directions. The entire SAD calculation stage is divided into two sub-stages: operand filling and pipeline calculation.

[0083] (1) Operand filling stage:

[0084] clk1~clk4: such as Figure 5 As shown in part (b), the cache controller retrieves the three numbers from the sub-window of Figure ① on the left and multicasts them to... Figure 6 In the array, PE00-PE03, PE10-PE13, and PE20-PE23 (multicast 43 to PE00-PE03, 87 to PE10-PE13, and 34 to PE20-PE23); the buffer controller sequentially retrieves the data from the four sub-windows in Figure ① on the right, and sends the data from the four sub-windows to the 12 PEs through the horizontal data transmission path of the array (send 88 to PE03, 59 to PE13, 88 to PE23, 1 to PE02, 45 to PE12, 6 to PE22, 42 to PE01, 58 to PE11, 14 to PE21, 69 to PE00, 72 to PE10, and 0 to PE20).

[0085] Pipeline calculation stage:

[0086] clk5: Each PE calculates the difference between two operands in the register and takes the absolute value. Taking PE00 as an example, it calculates the AD value (26) of the two operands (43 and 69) and stores it.

[0087] clk6: Updates the operands in each PE register, calculates the AD values ​​of the updated two operands, and adds the AD values ​​generated at clk5 column by column. Specifically, PE03 operand is updated to 1, PE13 operand to 45, PE23 operand to 6, PE02 operand to 42, PE12 operand to 58, PE22 operand to 14, PE01 operand to 69, PE11 operand to 72, PE21 operand to 0, PE00 operand to 55, PE10 operand to 80, and PE20 operand to 87. The four sub-windows in right figure ② have reusable values ​​with the four sub-windows in right figure ①. Therefore, only the values ​​of the rightmost sub-window (55, 80, 87) are retrieved from the buffer (internal structure of the PE) and sent to PE00, PE10, and PE20. The operands in the other three sub-windows are passed horizontally from the left PE to the right PE. Taking PE00 as an example, the AD values ​​(43) of the two operands (98 and 55) in the register are calculated and stored. The addition of AD values ​​is completed in two steps: taking the first column as an example, the first step is to add the AD values ​​of PE00 and PE10 and generate an intermediate result p, and the second step is to add the AD value of PE20 and the intermediate result p. The first step of the above process is completed at clk6. The AD value (15) of PE10 is passed to PE00 and added to the AD value (26) of PE00, and the intermediate result (41) is temporarily stored. The process of the remaining columns is the same as that of the first column.

[0088] clk7: Updates the operands in each PE register, calculates the AD values ​​of the two updated operands, and completes the second step of adding the AD values ​​generated at clk5, while simultaneously completing the first step of adding the AD values ​​generated at clk6. The update of PE operands and the calculation of operand AD values ​​are similar to the above process. Specifically, taking the first column as an example, PE20 passes the AD value (34) of the operands (34 and 0) generated at clk5 to PE00, and adds it with the intermediate result (41) at clk6 to obtain PSUM1_1 (75). PE10 passes the AD value (18) of the operands (98 and 80) generated at clk6 to PE00, and adds it with the AD value (43) of the operands (98 and 55) generated by PE00 at clk6 to produce the intermediate result (61). The remaining columns are the same as the first column.

[0089] clk8: Updates the operands in each PE register, calculates the AD values ​​of the updated two operands, and completes the second step of adding the AD values ​​generated at clk6, while simultaneously completing the first step of adding the AD values ​​generated at clk7. The update of PE operands and the calculation of operand AD values ​​are similar to the above process. Specifically, taking the first column as an example, PE20 passes the AD value (48) of the operands (39 and 87) generated at clk6 to PE00, and adds it with the intermediate result (61) at clk7 to obtain PSUM2_1 (109). PE10 passes the AD value (89) of the operands (90 and 1) generated at clk7 to PE00, and adds it with the AD value (12) of the operands (44 and 56) generated by PE00 at clk7 to produce the intermediate result (101). The remaining columns are the same as the first column.

[0090] clk9: Completes the second step of adding the AD values ​​generated at clk7, and simultaneously adds PSUM1_1~4 and PSUM2_1~4 respectively. Specifically, taking the first column as an example, PE20 passes the AD value (0) of the operands (45 and 45) at clk7 to PE00, and adds it to the intermediate result (101) at clk8 to obtain PSUM3_1 (101). In PE00, PSUM1_1 (75) is added to PSUM2_1 (109) to obtain the intermediate result q (184). The remaining columns are the same as the first column.

[0091] clk10: Adds the intermediate result q to PSUM3_1 to produce the final SUM value. Specifically, taking the first column as an example, in PE00, PSUM3_1 (101) is added to the intermediate result q (184) to obtain the final SUM value (285). The remaining columns are the same as the first column.

[0092] In this model, the PSUM value is passed vertically upwards between PEs, while the data in the right figure and the SAD calculation results generated by the Ultra PE and Standard PE rows are passed horizontally between PEs. This accelerates the calculation of the SAD matching cost and reduces the time wasted by repeatedly reading data. This PE connection method is more conducive to the implementation of a multi-stage pipeline design for the SAD matching cost, achieving the real-time requirements for data transmission and calculation.

[0093] Optionally, the PE array includes one or more of the following types of PEs:

[0094] Ultra PE is used to perform the absolute difference of two operands and the accumulation of partial sums during the SAD matching cost calculation process.

[0095] Standard PE is used to perform the subtraction and absolute value calculation of two operands, as well as the accumulation operation of partial sums, during the SAD matching cost calculation process.

[0096] Lite PE is used to perform the operation of taking the absolute value of the difference between two operands during the SAD matching cost calculation process;

[0097] Ultra PE has greater computing resources than Standard PE.

[0098] Specifically, when arranging the PE array, the PE array includes one or more of the three types of PE. For example, the PE array can be generated entirely from Ultra PE; it can also be generated from Ultra PE and Standard PE; or it can be generated from Ultra PE, Standard PE and Lite PE together.

[0099] Figure 7 This is a diagram of the internal structure of the Ultra PE provided by the present invention. Figure 7 As can be seen, the Ultra PE includes the following components: AD value accumulation units 701-705, which are used to accumulate intermediate results (AD values) transmitted from PEs in the vertical direction and generate PSUM, which is then transmitted to other PEs via data interconnection; AD calculation unit 706, which is used to calculate the difference between two operands and take the absolute value (hereinafter referred to as AD value); PSUM cache unit 707, which is used to temporarily cache historical PSUM locally during time-sharing multiplexing; PSUM accumulation unit 708; matching cost cache unit 709; operand cache unit 710, which is used to temporarily store operands during time-sharing multiplexing.

[0100] It should be noted that Ultra PE has the most computing resources, followed by Standard PE, and Lite PE has the least. Standard PE contains... Figure 7 The AD value accumulation units are 701-703; the AD calculation unit is 706; the PSUM cache unit is 707; the PSUM accumulation unit is 708; the matching cost cache unit is 709; and the operand cache unit is 710. Lite PE only contains... Figure 6 The AD calculation unit 706 and operand cache unit 710 are included.

[0101] Specifically, the AD value accumulation units 701-705 include an adder, two registers (A and B), a MUX, and a DEMUX. Specifically, operand 1 of the adder originates from the vertical data bus, transmitted from other PEs; operand 2 is the historical accumulation result or 0; after completing the accumulation of one PSAD, operand 2 is set to zero to proceed to the next PSAD calculation; the intermediate accumulated value is registered in A and B, which form a FIFO with a depth of 2; the MUX can choose to retrieve the accumulated value from either A or B (the register at the top in the diagram is B, and the register at the bottom is A); retrieving the intermediate accumulated value from B can achieve the effect of registering one clock cycle (delaying one preset clock cycle); the outputs of the five AD value accumulation units are connected to the MUX, which selects the valid PSUM and sends it to the PSUM buffer unit 707 and the PSUM accumulation unit 708 respectively.

[0102] The AD calculation unit 706 takes left and right image pixel data as input and outputs through a FIFO consisting of three registers. The MUX can choose to obtain AD values ​​from the three registers to achieve the effect of one beat (delayed by a preset clock cycle) or two beats.

[0103] The PSUM accumulation unit 708 has three operands: the current PSUM value, the historical PSUM value, and the historical matching cost. The PSUM value is generated by the AD accumulation units 701-705 and passed through the MUX. The historical PSUM value is cached in the PSUM value cache 707, and the historical matching cost is cached in the matching cost cache unit 709.

[0104] Those skilled in the art should understand that the number of AD value accumulation units, AD calculation units, etc., and the number of registers in the figure are not limiting and can be adjusted as needed.

[0105] Based on the different computing resource requirements of SAD multi-stage pipeline computing, different PE structures were designed, which effectively improved array utilization, saved FPGA resources, and reduced FPGA power consumption.

[0106] Optionally, each column in the PE array can be configured to include one or more PUs, which are used to perform SAD matching cost calculation operations with a specified window size.

[0107] Specifically, in order to perform as many parallax parallel calculations as possible in space, and also to be compatible with different matching window sizes and different parallax search depths, each column of the PE array can be configured as one or more PUs of a certain size. For example, if each column of the PE array has 10 PEs, it can be configured as 3 PUs to calculate the 3×3 window matching cost and complete three SAD matching cost calculations in parallel, or configured as 2 PUs to calculate the 5×5 window matching cost and complete two SAD matching cost calculations in parallel, or configured as 1 PU to calculate the 9×9 window matching cost and complete one SAD matching cost calculation.

[0108] Optionally, each column of the PE array has 19 PEs, which can be configured into 22 combinations of computational units as shown in Table 1. The following configuration combinations achieve compatibility with different window sizes. It should be noted that the numbers in the table represent the number of computational units capable of calculating the left and right window matching costs for the corresponding window size. For example, combination 1 means that each column of PEs can be configured with 5 computational units for calculating the left and right window matching costs for a 3×3 window.

[0109] Table 1: Configuration combinations of single-column PEs in a PE array

[0110]

[0111] Building upon this, by combining time-sharing multiplexing, it can accommodate larger disparity search depths. For example, each column of the PE array has 19 PEs, for a total of 25 columns. Each column of PEs can be configured as 5 PUs for calculating the matching cost of left and right windows of size 3×3. Thus, 25 columns of PEs can provide the parallel processing power of 125 3×3 windows. When the disparity search depth is less than or equal to 125, the array can achieve full disparity parallel computation; when the disparity search depth is greater than 125, time-sharing multiplexing is used for computation.

[0112] The embodiments of the present invention provide three time-division multiplexing configurations, namely time-division multiplexing 2 times, 4 times and 8 times. Table 2 shows the maximum search depth that this array can achieve under different time-division configurations.

[0113] Table 2: Theoretical Maximum Search Depth under Different Window Widths and Reuse Counts

[0114]

[0115] By following the above operations, the device can be compatible with different window sizes and different search depths, enabling the configuration of window size and search depth. At the same time, the maximum parallel design also enables the device to meet high real-time requirements.

[0116] Optionally, in this PU, the PE in the first row is either Ultra PE or Standard PE.

[0117] Specifically, in the process of calculating the matching cost, intermediate results are passed upwards in the vertical direction. Therefore, each PU accumulates the final result in the first row, so the first row can use Ultra PE or Standard PE.

[0118] Furthermore, when calculating SAD in the entire PE array, considering both resource consumption and compatibility with different window sizes, all rows containing adders should be Ultra PE or Standard PE. When there are many adders required (e.g., 5), only Ultra PE can be used; rows that only perform AD value calculations can use Lite PE.

[0119] Based on this design concept, for example, in a 19-row, 25-column PE array, each column of PEs can be configured as one or more PUs to perform SAD calculations. Ultra PEs are located in rows 0 and 9, Standard PEs in rows 2, 4, 6, 8, 11, 13, and 15, and Lite PEs in rows 1, 3, 5, 7, 10, 12, 14, 16, 17, and 18. This arrangement design achieves optimal resource allocation and avoids design redundancy.

[0120] Placing an Ultra PE or Standard PE on top of the PU can achieve the advantages of low resource consumption and low device power consumption.

[0121] The methods and apparatus provided in the various embodiments of this application are based on the same concept. Since the methods and apparatus solve problems in similar ways, their implementations can be referred to each other, and repeated details will not be repeated.

[0122] Figure 8 The flowchart of the configurable real-time parallax point cloud computing method provided by the present invention is shown below. Figure 8 As shown, the method includes the following steps:

[0123] Step 800: The configuration parsing module parses the received configuration information and generates corresponding control signals, which are then input to the cache controller, PE array, result shaping module, and minimum value search module, respectively.

[0124] Step 801: The cache controller controls the image cache unit to output image window data corresponding to one or more binocular image data according to the control signal transmitted by the configuration parsing module and the specified window size and sliding window order.

[0125] Step 802: The PE array generates several PUs with a specified structure according to the control signals transmitted by the configuration parsing module, and processes the input image window data based on the several PUs with the specified structure to obtain the SAD matching cost calculation result corresponding to the image window data.

[0126] Step 803: The result shaping module adds fields to the SAD matching cost calculation result output by the PE array according to the control signal transmitted by the configuration parsing module.

[0127] Step 804: The minimum value search module searches for the minimum value step by step in the SAD matching cost calculation result after adding fields, based on the control signals and minimum value search algorithm transmitted by the configuration parsing module, and outputs the disparity value corresponding to the minimum matching cost.

[0128] By configuring the parsing module, different matching parameters can be adapted; the SAD matching cost is calculated through the PE array pipeline structure to ensure high real-time performance, thereby overcoming the shortcomings of existing technologies that cannot simultaneously achieve high real-time performance and adapt to different matching information.

[0129] Optionally, the configuration information for the real-time parallax point cloud computing method includes image resolution, matching window size, parallax search depth, number of channels for binocular image data, and PE working mode.

[0130] Specifically, image resolutions support 480p, 720p, 1080p, etc.; matching window sizes include 3×3, 5×5, 13×13, etc.; the disparity search depth depends on the application scenario and is specified by the user; the binocular image data contains the data streams of the left and right images, referred to as left-right data stream pairs, and the number of paths in the binocular image data refers to the number of left-right data stream pairs; the PE working modes include Ultra PE, Standard PE, and Lite PE. During the parallel calculation of SAD matching costs, rows in the PE array that require adders use Ultra PE and Standard PE (Ultra PE is used when there are many adders), while rows that only perform AD value calculations use Lite PE.

[0131] This configuration information can be passed and parsed through the configuration parsing module, thereby enabling flexible adaptation to different matching parameters.

[0132] Optionally, the methods for determining configuration information include:

[0133] Determine whether the performance metrics for single data stream or multiple data stream are met;

[0134] Computational resources are allocated based on the number of allocable unit computing resources in the PE array, the disparity search depth corresponding to each data stream, and the video frame rate corresponding to each data stream.

[0135] Configuration information is generated based on the allocated computing resources.

[0136] Specifically, a data stream refers to a pair of left and right eye video data streams, while multiple data streams refer to multiple pairs of left and right eye video data streams.

[0137] A unit of computational resources refers to a unit consisting of several PEs that can be used to process the matching cost of a window of a certain size. For example, for a 3×3 window, its computational unit is a 3x1 matrix of PEs, containing 3 PEs.

[0138] Figure 2 Each module can be configured for a multi-data-stream sharing mode. The image buffer unit 200 can support the shaping of multiple data streams with different resolutions. The PE array uses a resource-aware configuration information generation method to spatially allocate data streams according to data stream parameters and frame rate requirements, thereby improving the utilization of computing resources.

[0139] When allocating computing units to a data stream, the configuration generation requirements are checked first, i.e., whether it is a multi-data stream. In the case of a multi-data stream, the architecture computing power is checked to see if it meets the multi-data stream performance requirements (i.e., meets or is close to the frame rate requirements under the specified window width and parallax search depth) based on the multi-data stream configuration space of this design. If it is a multi-data stream and meets the performance requirements, the configuration generation process is entered; otherwise, the single-data stream configuration process is entered or an error is reported.

[0140] In the process of determining resource allocation, computing resources are allocated based on the number of allocable unit computing resources in the PE array, the disparity search depth corresponding to each data stream, and the frame rate requirement corresponding to each data stream. For example, after allocating resources from the configuration resource space to the data streams to be allocated, the allocation of resources to each data stream is dynamically adjusted based on the difference between the proportion of computing resources already allocated to the data streams to be allocated and the proportion of computing resources required by the data streams to be allocated. The computing resources required by the data streams are characterized by the disparity search depth and the video frame rate.

[0141] The above method can allocate computing resources to each data stream based on the availability of array computing resources and the performance requirements of each data stream, thereby achieving the effect of making full use of computing resources and balancing the performance indicators of different data streams.

[0142] Optionally, when there are two data streams, computing resources are allocated based on the number of allocable unit computing resources in the PE array, the disparity search depth corresponding to each data stream, and the video frame rate corresponding to each data stream, including:

[0143] Determine the remaining allocatable computing resources in the PE array after allocating one unit of computing resource to each data stream;

[0144] If the remaining allocable computing resources can provide at least one unit of computing resources for each data stream and the first condition is met, then continue to allocate one unit of computing resources for each data stream.

[0145] If the remaining allocable computing resources can provide at least one unit of computing resources for each data stream, but the first condition is not met, then each data stream is allocated a unit of computing resources separately according to the relationship between the first and second values.

[0146] The first value is determined based on the parallax search depth and video frame rate corresponding to each data stream, the second value is determined based on the number of unit computing resources currently allocated to each data stream, and the first condition is determined based on the first value, the second value, and a preset threshold.

[0147] Specifically, Figure 9 This is a schematic diagram of the resource-aware configuration generation process provided by the present invention. As can be seen from the diagram, taking the allocation of resources to two data streams A and B as an example, where… As the first condition, δ is an empirically set value, a small quantity, such as 0.2 or 0.5. The first value, The second value represents I, which indicates the number of allocated computing resources, D, which indicates the search depth required by the data stream, and F, which indicates the frame rate required by the data stream. The configuration generation process is as follows: First, allocate one unit of computing resource to each data stream. Then, check if all available computing resources exist for data streams A and B in the array. If so, proceed with further checks. If the above formula is true, then continue to allocate unit calculation resources to both A and B. If it is not true, then check... and The size relationship, if If A is insufficient, allocate unit computing resources specifically for A; otherwise, allocate unit computing resources specifically for B and continue checking. The configuration information will be output once the configuration generation is complete.

[0148] Using the above method, resources can be dynamically allocated to each data stream, resulting in better resource allocation and higher utilization.

[0149] Optionally, the resource allocation method further includes:

[0150] If the remaining allocable computing resources can only provide one unit of computing resources for the target data stream, then all remaining allocable computing resources will be allocated to the target data stream.

[0151] Specifically, when allocating resources for two data streams, first, one unit of computing resource is allocated to each data stream. Then, it is checked whether there are allocable computing resources for both data streams A and B in the array. If the above condition is not met, it is checked whether there are any remaining units of computing resources that can be allocated to data stream A. In this case, data stream A is the target data stream. If there are any remaining units of computing resources, all remaining computing resources are allocated to data stream A. If there are no remaining units of computing resources that can be allocated to data stream A, it is checked whether there are any remaining units of computing resources that can be allocated to data stream B. In this case, data stream B is the target data stream. If there are any remaining units of computing resources, all remaining computing resources are allocated to data stream B, the configuration ends, and the configuration information is output. If there are no remaining computing resources that can be allocated to any data stream, the configuration ends, and the configuration information is output.

[0152] This allocation method allows for the allocation of resources to a single data stream when computing resources are insufficient to allocate two data streams simultaneously, but sufficient to allocate one data stream, thereby maximizing resource utilization.

[0153] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0154] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-executable instructions. These computer-executable instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0155] These processor-executable instructions may also be stored in a processor-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the processor-readable memory produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These processors can execute instructions that can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0157] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A configurable real-time parallax point cloud computing device, characterized in that, include: Image caching unit, cache controller, processing unit PE array, result shaping module, minimum value search module, and configuration parsing module; The image caching unit is connected to the caching controller and is used to shape the cached binocular image data according to a specified window size and sliding window order under the control of the caching controller, and then output the image window data to the caching controller. The cache controller is connected to the configuration parsing module and the PE array respectively, and is used to control the image cache unit to output image window data according to the control signal transmitted by the configuration parsing module. The image window data is distributed to the PEs in the PE array by the cache controller. The PE array is connected to the configuration parsing module and the result shaping module respectively. It is used to generate a number of algorithm processing units (PUs) with a specified structure according to the control signal transmitted by the configuration parsing module, and process the input image window data based on the number of PUs with the specified structure to obtain the SAD matching cost calculation result and output it to the result shaping module. The result shaping module is connected to the configuration parsing module and the minimum value search module respectively, and is used to add fields to the input SAD matching cost calculation result according to the control signal transmitted by the configuration parsing module and then output it to the minimum value search module. The minimum value search module is connected to the configuration parsing module and is used to search for the minimum value step by step based on the control signal and minimum value search algorithm transmitted by the configuration parsing module, and output the disparity value corresponding to the minimum matching cost. The configuration parsing module is used to parse the received configuration information and generate corresponding control signals, which are then input to the cache controller, the PE array, the result shaping module, and the minimum value search module.

2. The configurable real-time parallax point cloud computing device according to claim 1, characterized in that, The PEs in the PE array are interconnected vertically and horizontally, with intermediate results being passed in the vertical direction and operands and final matching costs being passed in the horizontal direction.

3. The configurable real-time parallax point cloud computing device according to claim 2, characterized in that, The PE array includes one or more of the following types of PE: Ultra PE is used to perform the absolute difference of two operands and the accumulation of partial sums during the SAD matching cost calculation process. Standard PE is used to perform the subtraction and absolute value calculation of two operands, as well as the accumulation operation of partial sums, during the SAD matching cost calculation process. Lite PE is used to perform the operation of taking the absolute value of the difference between two operands during the SAD matching cost calculation process; The computing resources corresponding to the Ultra PE are greater than those of the Standard PE.

4. The configurable real-time parallax point cloud computing device according to claim 3, characterized in that, Each column in the PE array can be configured to include one or more PUs, which are used to perform SAD matching cost calculation operations with a specified window size.

5. The configurable real-time parallax point cloud computing device according to claim 4, characterized in that, In the PU, the PE in the first row is either the Ultra PE or the Standard PE.

6. A configurable real-time parallax cloud computing method based on a configurable real-time parallax cloud computing device as described in any one of claims 1 to 5, characterized in that, The method includes: The configuration parsing module parses the received configuration information and generates corresponding control signals, which are then input to the cache controller, the PE array, the result shaping module, and the minimum value search module, respectively. The cache controller, based on the control signals transmitted by the configuration parsing module, controls the image cache unit to output image window data corresponding to one or more channels of binocular image data according to the specified window size and sliding window order; The PE array generates several PUs with a specified structure according to the control signals transmitted by the configuration parsing module, and processes the input image window data based on the several PUs with the specified structure to obtain the SAD matching cost calculation result corresponding to the image window data. The result shaping module adds fields to the SAD matching cost calculation result output by the PE array according to the control signal transmitted by the configuration parsing module; The minimum value search module searches for the minimum value step by step in the SAD matching cost calculation result after adding fields, based on the control signals and minimum value search algorithm transmitted by the configuration parsing module, and outputs the disparity value corresponding to the minimum matching cost.

7. The configurable real-time parallax point cloud computing method according to claim 6, characterized in that, The configuration information includes: Image resolution, matching window size, parallax search depth, number of channels for binocular image data, and PE working mode.

8. The configurable real-time parallax point cloud computing method according to claim 7, characterized in that, The methods for determining the configuration information include: Determine whether the performance metrics for a single data stream or multiple data streams are met; Computational resources are allocated based on the number of allocable unit computing resources in the PE array, the disparity search depth corresponding to each data stream, and the video frame rate corresponding to each data stream. Configuration information is generated based on the allocated computing resources.

9. The configurable real-time parallax point cloud computing method according to claim 8, characterized in that, When there are two data streams, the allocation of computing resources based on the number of allocable unit computing resources in the PE array, the disparity search depth corresponding to each data stream, and the video frame rate corresponding to each data stream includes: After allocating one unit of computing resource to each data stream, the remaining allocable computing resources in the PE array are determined; If the remaining allocable computing resources can provide at least one unit of computing resources for each data stream and the first condition is met, then continue to allocate one unit of computing resources for each data stream. If the remaining allocable computing resources can provide at least one unit of computing resources for each data stream, but the first condition is not met, then according to the relationship between the first and second values, a unit of computing resources is allocated to each data stream individually. The first value is determined based on the parallax search depth and video frame rate corresponding to each data stream, the second value is determined based on the number of unit computing resources currently allocated to each data stream, and the first condition is determined based on the first value, the second value, and a preset threshold.

10. The configurable real-time parallax point cloud computing method according to claim 9, characterized in that, The method further includes: If the remaining allocable computing resources can only provide one unit of computing resources for the target data stream, then all the remaining allocable computing resources are allocated to the target data stream.

Images