Neural network accelerators for image processing and 3D reconstruction methods

Abstract

This invention provides a neural network accelerator device for image processing. The device includes: a multiply-accumulate operation module configured with multiple matrix operation units for performing continuous multiply-accumulate operations; a data storage module configured with a multi-level storage structure for hierarchical multi-level storage and data sharing; a data management module for retrieving computational data and allocating storage space; a general-purpose processor for solving extrinsic parameters; and an auxiliary operation module for performing other indivisible operations besides multiply-accumulate and extrinsic parameter matrix solving. This invention distributes the three computationally demanding operations—feature point matching, depth prediction, and depth fusion—to three matrix operation units and rationally allocates computation time and operator ratios, ensuring that the operations of each unit do not interfere with each other, achieving a three-stage pipeline parallel processing effect, and improving the accelerator's efficiency and compatibility for image processing.

Description

Technical Field

[0001] This invention relates to the field of computers, more specifically to the field of neural network accelerators, and more specifically to an accelerator device for image processing and a three-dimensional reconstruction acceleration control method based thereon. Background Technology

[0002] The development of machine vision involves neural networks processing massive amounts of data. With the integration of big data processing technologies and deep learning algorithms, machine vision increasingly requires higher-performance processor chips to process large amounts of image information, especially in 3D reconstruction. In embedded systems for image processing, balancing system performance and power consumption is a key consideration. However, traditional central processing units (CPUs) struggle with image data processing due to their lack of instruction parallelism; while traditional image processors (EPPs), due to their versatility, consume significant power when running neural network models with large data loads.

[0003] Therefore, neural network accelerators, with their high performance and low power consumption for neural network operations, are widely embedded in terminal devices for parallel image processing and have become a typical accelerator for accelerating image processing algorithms on the terminal. With the emergence of neural network algorithms with different structures, various neural network accelerators designed to accelerate these algorithms have been developed. However, because neural network accelerators often require adaptive optimization for specific network models, accelerators for different neural networks present different challenges.

[0004] Among them, neural network accelerators for 3D reconstruction of consecutive frame images require adaptive design of accelerator structure and data processing sequence due to their large computational load and high parallelism requirements. In existing technologies, 3D reconstruction neural networks first establish feature point matching between multi-view images to find and match the same spatial point in different view images; then, after obtaining the disparity of the same spatial point in two images, they predict the depth of the spatial point through geometric relationships to reconstruct a dense point cloud; finally, they perform depth fusion to generate a surface model of the 3D object. This type of reconstruction involves massive amounts of data and high computational complexity, making it difficult to deploy on edge devices and failing to meet real-time requirements. Therefore, to address the high computational cost of 3D reconstruction while meeting the requirements for real-time model operation, it is necessary to design dedicated hardware accelerator devices and algorithm schemes. Summary of the Invention

[0005] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a new neural network accelerator device and a three-dimensional reconstruction method for image processing.

[0006] According to a first aspect of the present invention, a neural network accelerator device for image processing is provided, the device comprising: a multiply-accumulate operation module configured with a plurality of matrix operation units for performing continuous multiply-accumulate operations; a data storage module configured with a multi-level storage structure for hierarchical multi-level storage of data and data sharing; a data management module for retrieving operation data and allocating storage space; a general-purpose processor for performing extrinsic parameter matrix solving operations; and an auxiliary operation module for performing other non-decomposable operations besides multiply-accumulate operations and extrinsic parameter matrix solving operations.

[0007] Preferably, the multiply-accumulate operation module includes three matrix operation units, each matrix operation unit is configured with a corresponding data cache unit, wherein each matrix operation unit is used to perform multiplication operations, and each data cache unit is used to directly exchange data with its corresponding matrix operation unit.

[0008] Preferably, each matrix unit includes multiple processing units and multiple array registers, each processing unit includes at least one multiplier and one adder, and the multiple array registers are used to transfer intermediate data processed by the processing unit.

[0009] Preferably, the multiply-accumulate operation module, the general-purpose processor, and the auxiliary operation module can be configured in a pipelined mode.

[0010] Preferably, the data storage module includes a secondary cache and main memory, wherein: the secondary cache is directly connected to each data cache unit in the multiply-accumulate module and is used to store data that cannot be stored in each data cache unit; the main memory is directly connected to the secondary cache and is used to store large-scale data that cannot be stored in the secondary cache.

[0011] Preferably, the data management module includes a register unit, a selector unit, a DMA controller unit, and a data temporary storage unit, wherein: the register unit is used for transferring intermediate data; the selector unit is used for selecting the plurality of matrix units to realize multi-stage pipeline operation; the DMA controller unit is used for arranging and transferring image data on the DMA path; and the data temporary storage unit is used for temporarily storing transferred data during peak data transmission periods.

[0012] Preferably, the general-purpose processor is configured to solve polynomials, solve Jacobian matrices, and calculate the extrinsic Rt rotation matrix at adjacent frame times of the image.

[0013] Preferably, the auxiliary operation module includes: a logic operation unit for performing various logic operations, including at least AND, OR, NOT, XOR, comparison, and / or shift logic operations; an arithmetic operation unit for performing operations on small-scale data that cannot be decomposed, including at least addition, subtraction, multiplication, division, squaring, and / or square root arithmetic operations; an activation function operation unit for performing activation operations on data, including at least ReLU activation function, binary activation function, and / or, and Sigmoid activation function activation operations; a data sorting unit for calling data from the data management module to perform data sorting operations, including sorting by finding the maximum value, sorting by finding the minimum value, and / or, and data sorting; a pooling operation unit for assisting the matrix operation unit in performing pooling calculations in the neural network; and an upsampling unit for assisting the matrix operation unit in performing upsampling calculations in the neural network.

[0014] According to a second aspect of the present invention, a three-dimensional reconstruction method based on the apparatus described in the first aspect of the present invention is provided. The method includes configuring the apparatus in a multi-stage pipeline mode, configuring multiple matrix operation units in the multiply-accumulate operation module to be responsible for feature point matching, multiple matrix operation units to be responsible for depth prediction, and multiple matrix operation units to be responsible for depth fusion. After acquiring RGB information of consecutive frames of images and camera correction information collected by a camera peripheral, the following steps are performed for each frame of image: S1, the matrix operation unit responsible for feature point matching, the data management module, and the general processor perform feature point matching operations based on the image RGB information and the camera correction information, and transmit the feature point matching results to the matrix operation unit responsible for depth prediction and the matrix operation unit responsible for depth fusion; S2, the matrix operation unit responsible for depth prediction, the data management module, and the auxiliary operation module perform depth prediction operations based on the RGB image information and the feature point matching results, and transmit the depth prediction results to the matrix operation unit responsible for depth fusion; S3, the matrix operation unit responsible for depth fusion, the data management module, and the auxiliary operation module perform depth fusion operations based on the RGB image information, the feature point matching results, and the depth prediction results to obtain three-dimensional point cloud information.

[0015] Preferably, the matrix units are allocated according to the time cycles required for feature point matching, depth prediction, and deep fusion, so as to balance the clock cycles of each node in the pipeline formed by feature point matching, depth prediction, and deep fusion.

[0016] Preferably, after obtaining the RGB information of the continuous frame images captured by the camera peripheral, the data management module calls it to store it in the main memory.

[0017] Preferably, after receiving the signal that the previous 3D reconstruction operation is complete, the data management module copies the RGB information of the current frame image from the main memory to the secondary cache.

[0018] Preferably, step S1 includes: S11, the data management module copies the RGB information of the current frame image from the secondary cache to the data cache unit corresponding to the matrix unit responsible for feature point matching; S12, after receiving the signal that the matrix unit responsible for feature point matching has completed the previous feature point matching, the module transmits the RGB information of the current frame image and the previous frame image in the corresponding data cache unit to the matrix unit responsible for feature point matching and deletes the RGB information of the frame before the previous one stored in the data cache unit; S13, the matrix unit responsible for feature point matching performs feature point matching based on the RGB information of the current frame image and the previous frame image in its corresponding data cache unit, and after the feature point matching is completed, stores the matched pixels in the corresponding data cache unit and sends them to the data cache unit. The data management module sends a feature point matching completion signal; S14, after receiving the feature point matching completion signal of the matrix unit responsible for feature point matching for the current frame image and the Rt rotation matrix calculation completion signal of the general processor, the data management module transmits the matching pixels stored in the data cache unit and the camera correction information in the main memory to the general processor; S15, the general processor calculates the external parameter Rt rotation matrix based on the matching pixels and camera correction information transmitted by the data management module, and sends an Rt rotation matrix calculation completion signal to the data management module after the calculation is completed; S16, after receiving the Rt rotation matrix calculation completion signal, the data management module stores the calculated external parameter Rt rotation matrix information of the camera for the two consecutive frames into the secondary cache.

[0019] Preferably, step S2 includes: S21, the data management module copies the RGB information and Rt rotation matrix information of the current frame image from the secondary cache to the data cache unit corresponding to the matrix unit responsible for depth prediction; S22, after receiving the signal that the matrix unit responsible for depth prediction has completed the previous depth prediction, the data management module transmits the RGB information of the current frame image and one or more historical frame images in the corresponding data cache unit to the matrix unit responsible for feature point matching and deletes the RGB information of unused images stored in the data cache unit; S23, the matrix unit responsible for depth prediction performs depth prediction based on the RGB information of the current frame image, Rt rotation matrix information, and RGB information of one or more historical frame images in its corresponding data cache unit, and sends a depth prediction completion signal to the data management module after the prediction is completed; S24, after receiving the depth prediction completion signal, the data management module stores the depth information and confidence information obtained from the depth prediction into the secondary cache.

[0020] Preferably, step S3 includes: S31, the data management module copies the RGB matrix information of the previous frame image, the RGB matrix information of the current frame image, the predicted depth information, the confidence information, and the Rt matrix information from the secondary cache to the data cache unit corresponding to the matrix unit responsible for deep fusion; S32, after receiving the signal that the previous deep fusion operation is completed, the data management module transmits the RGB matrix information of the previous frame image, the RGB matrix information of the current frame image, the predicted depth information, the confidence information, and the Rt matrix information in the corresponding cache unit to the matrix unit responsible for deep fusion; S33, based on the signal that the previous deep fusion operation is completed, the matrix unit responsible for deep fusion performs deep fusion on the RGB matrix information of the previous frame image, the RGB matrix information of the current frame image, the predicted depth information, the confidence information, and the Rt matrix information in the corresponding cache unit, and sends a deep fusion completion signal to the data management module before the fusion is completed; S34, after receiving the deep fusion completion signal, the data management module stores the 3D point cloud information obtained by deep fusion into the main memory.

[0021] Compared with existing technologies, the advantages of this invention are as follows: This invention distributes the three computationally intensive operations—feature point matching, depth prediction, and depth fusion—to three matrix operation units. By rationally allocating the computation time and operator ratio of each matrix operation unit, the operations between the units can be made independent of each other, achieving the parallel processing effect of a three-stage pipeline. This parallel processing architecture improves the efficiency and compatibility of the accelerator for image processing in 3D reconstruction tasks. Attached Figure Description

[0022] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0023] Figure 1 This is a schematic diagram of the frame structure of an accelerator device for image processing according to an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of a specific structure of an accelerator device for an image processor according to an embodiment of the present invention;

[0025] Figure 3 This is a schematic diagram illustrating the three-dimensional reconstruction principle of an accelerator device based on an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of the three-dimensional reconstruction process according to an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of the time flow of three-dimensional reconstruction according to an embodiment of the present invention.

[0028] Figure 6This is a schematic diagram of the computational load allocation during the three-dimensional reconstruction process according to an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0030] To address the problems of existing 3D reconstruction neural network accelerators, this invention, referencing the scalability of multiply-accumulate arrays and the parallelism of pipelined processing techniques, proposes an accelerator device for image processing based on multiply-accumulate arrays. This device enables parallel processing of feature point matching, depth prediction, and depth fusion in 3D reconstruction, achieving a pipelined processing architecture. It is applicable not only to 3D reconstruction but also to other visual image processing fields.

[0031] To better understand the present invention, the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0032] According to one embodiment of the present invention, such as Figure 1 As shown, an accelerator device for image processing according to the present invention includes: a multiply-accumulate operation module configured with multiple matrix operation units for performing continuous multiply-accumulate operations; a data storage module configured with a multi-level storage structure for hierarchical multi-level storage and data sharing; a data management module for retrieving operation data and allocating storage space; a general-purpose processor for performing extrinsic parameter matrix solving operations; and an auxiliary operation module for performing other indivisible operations besides multiply-accumulate operations and extrinsic parameter matrix solving operations. The device of the present invention incorporates a multiply-accumulate operation module, a general-purpose processor, and an auxiliary operation module, which can be combined into a pipelined processing architecture, enabling the accelerator to not only perform parallel operations of various visual algorithms but also further improve the algorithm running speed. Furthermore, considering that the input data of the 3D reconstruction neural network is continuous frame image information collected by the camera peripheral, in order to avoid the situation where the internal storage space is insufficient due to excessive data volume, this invention, while meeting the basic data requirements required for the operation of the algorithm, divides the internal storage space of the accelerator into multi-level storage according to the data access speed, and rationally allocates the data storage space through the data management module to maximize the rational use of storage space and improve the operating efficiency of the accelerator.

[0033] Among them, still refer to Figure 1The multiply-accumulate operation module is used to implement large-scale continuous multiply-accumulate operations in the 3D reconstruction algorithm. The implemented multiply-accumulate operations include, but are not limited to, image convolution feature extraction, neural network convolution operations, decomposed deep fusion algorithms, absolute value operations, and square root operations. Each multiply-accumulate operation module includes multiple matrix operation units, and each matrix unit is configured with a corresponding data cache unit (in this embodiment, the data cache unit uses SRAM).

[0034] According to one embodiment of the present invention, such as Figure 2 As shown, it includes at least three matrix operation units and their matching SRAM. Each matrix operation unit can be composed of a systolic array, a parallel Booth algorithm multiplier array, a Wallace tree multiplier array, a lookup table multiplier array, or a designed application-specific integrated circuit (ASIC) capable of large-scale continuous multiplication and addition operations. The commonality of different matrix operation arrays is that they can process a certain number of multiplication operations in parallel or in a pipelined, time-division multiplexing manner within a unit cycle. Simultaneously, the multiply-accumulate array needs to possess a certain degree of scalability to improve the system's hardware computing power to meet the real-time requirements of different 3D reconstruction models. In this embodiment, a systolic array is used as the matrix operation unit for detailed description. Using other multiply-accumulate arrays achieves the same effect, but the external interface must be consistent to meet system compatibility requirements. Other arrays will not be described again. A systolic array includes multiple processing units (PEs) and several registers. Each Processing Array (PE) can consist of multipliers such as Booth multipliers, lookup table multipliers, and Wallace tree multipliers for multiplication operations, and adders such as pipelined adder trees and parallel adder arrays for accumulation operations. A processing unit includes, but is not limited to, one multiplier and one adder. Furthermore, during image data processing, the processing unit of the systolic array caches intermediate results in real time through registers and periodically outputs the operation results to the SRAM data cache unit according to the size of the input matrix.

[0035] According to one embodiment of the present invention, still referring to Figure 2The data storage module shown is used to store and share accelerator data. This module includes, but is not limited to, a multi-level storage structure consisting of a secondary cache and main memory (in this case, the SRAM in the multiply-accumulate operation module is equivalent to the fastest primary cache, followed by the secondary cache, and the main memory is the slowest). The secondary cache is directly connected to the SRAM data cache unit and is used to store data that cannot be stored in the SRAM data cache unit to achieve data sharing between matrix operation units. The main memory is directly connected to the secondary cache and is used to store data that cannot be stored in the secondary cache, continuous frame image information, and physical 3D point cloud information, etc. The SRAM data cache unit is a private cache unit for each module and is used for high-speed data exchange with fixed modules. The secondary cache and main memory are shared storage units. The secondary cache enables high-speed data sharing between modules; the main memory is used for storing large-scale data. This invention, by setting up multi-level storage, can meet the needs of both high system data access speed and large capacity.

[0036] According to one embodiment of the present invention, still referring to Figure 2 The data management module is used for retrieving computational data and allocating storage space. This data management module includes, but is not limited to, at least one register unit (a register unit may include multiple registers, referred to as the first register, second register, ..., Nth register), at least one gating unit (a gating unit may include multiple gatings, referred to as the first gating, second gating, ..., Nth gating), at least one DMA controller unit (a DMA controller unit may include multiple controllers, referred to as the first controller, second controller, ..., Nth controller), and at least one data buffer unit. The DMA controller unit is a direct memory access module used to alleviate the data transfer pressure on the general-purpose processor during peak data processing periods. The data management module is directly connected to the general-purpose processor, auxiliary computation modules, various levels of storage modules, and peripherals. Specifically, the addressing register module is used for moving small-scale intermediate computational data; the data gating is used to select data paths to facilitate multi-stage pipeline operations; the DMA controller unit is used for arranging and moving large-scale image matrix data along the DMA path; and the data buffer unit is used for temporarily storing moved data during peak data transfer periods.

[0037] According to one embodiment of the present invention, the operations performed by the general-purpose processor include, but are not limited to, solving polynomials, solving Jacobian matrices, and calculating the extrinsic rotation matrix Rt of the camera at adjacent frame times.

[0038] According to one embodiment of the present invention, the auxiliary operation module is used to perform operations that are difficult to decompose in the image processing neural network. The auxiliary operation module includes, but is not limited to, a logic operation unit, an arithmetic operation unit, an activation function operation unit, a data arrangement unit, a pooling operation unit, and an upsampling unit. The logic operation unit is used to perform various logic operations, including but not limited to a logical AND unit, a logical OR unit, a logical NOT unit, an XOR greater than unit, a comparison operation unit, and a shift operation unit, to perform AND, OR, NOT, XOR, comparison, and shift logic operations respectively. The arithmetic operation unit is used to perform small-scale operations that are difficult to decompose, including but not limited to an addition operation unit, a subtraction operation unit, a multiplication operation unit, and a division operation unit, to implement, but is not limited to, addition, subtraction, multiplication, division, square, and square root arithmetic operations. The activation function unit performs activation operations on the data, including but not limited to ReLU activation units, binary activation units, linear activation units, and Sigmoid activation units, to implement arithmetic operations using the 'ReLU function,' 'binary activation function,' 'linear activation function,' and 'Sigmoid activation function.' The data sorting unit sorts the data for computation by calling data from the data management module; its operations include, but are not limited to, finding the maximum and minimum values, and data sorting. The pooling operation unit and the upsampling unit assist the matrix operation unit in performing pooling and upsampling calculations in the neural network.

[0039] Based on the accelerator device proposed in this invention, this invention further proposes a 3D reconstruction method based on this device. In this method, the three computationally intensive operations of feature point matching, depth prediction, and depth fusion in 3D reconstruction are assigned to different matrix operation units (if there are three matrix operation units, they are used for feature point matching, depth prediction, and depth fusion respectively; if there are more than three matrix operation units, they can be divided into three parts). Data retrieval through the data management module can divide the 3D reconstruction process into a three-stage pipeline structure on the accelerator device of this invention, thereby improving the parallelism and computational efficiency of 3D reconstruction. In subsequent embodiments of this invention, the multiply-accumulate operation module includes three systolic arrays as an example, and the following is an example of systolic array 1 (corresponding to data cache SRAM1) performing feature point matching, systolic array 2 (corresponding to data cache SRAM2) performing depth prediction, and systolic array 3 (corresponding to data cache SRAM3) performing depth fusion. In summary, during feature point matching, pulsating array 1 receives the image's RGB matrix information, performs feature point matching, and after the calculation, the data management module inputs the matched pixels into the general-purpose processor. The general-purpose processor receives the matched pixels and correction information from the camera, calculates the Rt rotation matrix, and after the calculation, the data management module inputs the calculated Rt rotation matrix into pulsating array 2 (responsible for depth prediction) and pulsating array 3 (responsible for depth fusion), respectively. During depth prediction, pulsating array 2 receives the image's RGB matrix information and Rt rotation matrix information and performs depth prediction calculations. Simultaneously, for calculations that are difficult to decompose, the data management module delegates the calculation to the auxiliary calculation module. After the depth prediction calculation is completed, the data management module inputs the calculation results from pulsating array 2 and the auxiliary calculation module into pulsating array 3 (responsible for depth fusion). During depth fusion, pulsating array 3 receives the image's RGB matrix information, Rt rotation matrix information, depth information obtained from depth prediction, and confidence information, and performs point cloud fusion calculations of the image depth information. Simultaneously, for calculations that are difficult to decompose, the data management module delegates the calculation to the auxiliary calculation module. After the point cloud fusion operation is completed, the data management module stores the operation results of the pulsating array 3 and the auxiliary operation module in the main memory, and then outputs them to the screen peripheral. By reasonably allocating the proportion of operators in the three matrix operation units, this invention enables part 1 feature point matching, part 2 depth prediction, and part 3 depth fusion operations to be performed concurrently, achieving a three-stage pipeline effect.

[0040] To better understand the application of the accelerator device of the present invention in the field of three-dimensional reconstruction, the three-dimensional reconstruction process will be described in detail below with reference to the accompanying drawings.

[0041] like Figure 3As shown, taking an accelerator device containing three matrix operation units as an example, when used for 3D reconstruction, the working modes of each module are as follows:

[0042] The pulsating array 1 and SRAM cell 1 exchange data through data path 1. The exchanged data includes, but is not limited to, the RGB matrix information of the (n-1)th frame image, the RGB matrix information of the nth frame image, and the pixel points of the (n-1)th frame image and the nth frame image used for feature point matching. The exchanged data is used for multiplication and accumulation operations in feature point matching.

[0043] SRAM unit 1 exchanges data with the data storage module and the data management module (SRAM unit 1 can directly exchange data with the data storage module, or it can exchange data with the data storage module through the data management module). The exchanged data includes, but is not limited to, the RGB matrix information of the nth frame image and the pixels matched by the feature point matching operation between the (n-1)th frame image and the nth frame image.

[0044] The pulsating array 2 and SRAM cell 2 exchange data via data path 2. The exchanged data includes, but is not limited to, the RGB matrix information of the nth frame image, intermediate data for feature extraction operations of the nth frame image, the source image feature matrix, and the reference image feature matrix. The exchanged data is used for multiply-accumulate operations in depth prediction.

[0045] SRAM unit 2 exchanges data with the data storage module and the data management module (SRAM unit 2 can directly exchange data with the data storage module, or it can exchange data with the data storage module through the data management module). The exchanged data includes, but is not limited to, the RGB matrix information of the nth frame image, the Rt matrix information, the reference image feature matrix, and the Rt matrix information.

[0046] The pulsating array 3 and the SRAM cell 3 exchange data through data path 3. The exchanged data includes, but is not limited to, the RGB matrix of the (n-1)th frame image, the RGB matrix of the nth frame image, depth, confidence score, or Rt matrix. The exchanged data is used for multiplication and accumulation operations in depth fusion.

[0047] SRAM unit 3 exchanges data with the data storage module and the data management module (SRAM unit 3 can directly exchange data with the data storage module, or it can exchange data with the data storage module through the data management module). The exchanged data includes, but is not limited to, the RGB matrix information of the (n-1)th frame image, the RGB matrix information of the nth frame image, the predicted depth information, the confidence information, and the Rt matrix information.

[0048] There is a data path between the data storage module and the data management module. The data exchanged includes, but is not limited to, RGB information of continuous frame images, data required for feature point matching calculation, data required for depth prediction calculation, and data required for depth fusion calculation.

[0049] Inside the data management module, there is a data path between the register unit, selector unit, DMA controller unit, and data temporary storage unit. This data path is used for data temporary storage during peak data transmission periods (during peak data transmission periods, the data path is congested. To avoid operation pauses caused by data congestion, the completed data is temporarily stored in the data temporary storage module, and the data is transmitted out when the data path is less congested).

[0050] The general-purpose processor and the data management module exchange data via data path 4. The exchanged data includes, but is not limited to, instruction codes, data storage addresses, matched pixels (for Rt rotation matrix operations), camera correction information (for Rt rotation matrix operations), and camera extrinsic Rt rotation matrix information (data after Rt rotation matrix operations). This data is used for data handling control and for scientific calculations on small amounts of data.

[0051] The auxiliary calculation module and the data management module exchange data through data path 5. The exchanged data includes, but is not limited to, intermediate data (source image feature matrix, reference image feature matrix or Rt matrix) required for depth prediction that cannot be decomposed into multiply-accumulate operations, and intermediate data (RGB matrix of the (n-1)th frame image, RGB matrix of the nth frame image, depth, confidence or Rt matrix) required for depth fusion that cannot be decomposed into multiply-accumulate operations.

[0052] The data management module exchanges data with peripherals via data path 6. The exchanged data includes, but is not limited to, point cloud data and RGB matrix information of consecutive frame images. This is used for data exchange with peripherals. In the data management module, the register unit is used for temporary data storage during data transfer. The selector unit is used to select an idle module during the pipeline process and to perform data transfer.

[0053] According to one embodiment of the present invention, as follows Figure 3 When the accelerator device shown is used for 3D reconstruction, the process is as follows:

[0054] Step 1: The camera peripheral acquires RGB information of consecutive frames of images and stores the RGB information of consecutive frames of images into main memory through the call of the data management module.

[0055] Step 2: After receiving the signal that the previous 3D reconstruction operation is complete, the data management module copies the RGB information of the current frame image (assuming the current frame image is sorted by time as n, hereinafter referred to as the nth frame image) to the secondary buffer in the form of a matrix.

[0056] Part 1, Feature Point Matching:

[0057] Step 1: The data management module copies the RGB matrix information of the nth frame image into SRAM unit 1 and waits for the pulsating array 1 to complete the previous feature point matching operation.

[0058] Step 2: After receiving the signal indicating that the previous feature point matching operation of the pulsating array 1 has been completed, the data management module inputs the RGB matrix information of the (n-1)th frame image and the nth frame image in the SRAM unit 1 into the pulsating array 1 through the data path 1 to perform neural network operations for feature point matching; at the same time, it deletes all information of the (n-2)th frame image in the SRAM unit 1.

[0059] Step 3: After the pulsating array 1 completes the feature point matching operation, it caches the matching pixels of the two frames into SRAM unit 1, and at the same time sends a feature point matching operation completion signal to the data management module.

[0060] Step 4: After receiving the feature point matching operation completion signal of the pulsating array 1 and the Rt rotation matrix calculation completion signal of the general processor, the data management module inputs the matching pixel points in SRAM unit 1 and the camera correction information in main memory into the general processor through data path 4.

[0061] Step 5: After receiving the matching pixel points and camera correction information transmitted by the data management module, the general-purpose processor calculates the extrinsic rotation matrix Rt. After the Rt rotation matrix calculation is completed, the general-purpose processor sends an Rt rotation matrix calculation completion signal to the data management module.

[0062] Step 6: After receiving the signal that the Rt rotation matrix calculation is complete, the data management module caches the calculated external parameter Rt rotation matrix information of the camera for two consecutive frames (note: the (n-1)th frame and the nth frame) into the secondary buffer.

[0063] Part 1 is complete.

[0064] Part 2, Depth Prediction:

[0065] In step 1, the data management module copies the RGB matrix information and Rt matrix information of the nth frame image into SRAM cell 2, and waits for the pulsating array 2 to complete the previous depth prediction operation.

[0066] Step 2: After receiving the depth prediction completion signal from the previous pulse array 2, the data management module inputs the RGB matrix information of the nth frame image in SRAM cell 2 into the pulse array 2 to perform depth prediction initialization. At the same time, it deletes all information of the (n-2)th frame image in SRAM cell 2.

[0067] Step 3: The systolic array 2 performs depth prediction initialization. Initialization includes, but is not limited to, feature extraction of the nth frame image, generation of the source image feature matrix, and updating of the reference image feature matrix in the secondary buffer. The source image feature matrix is ​​the feature matrix of the nth frame image, and the reference image feature matrix includes, but is not limited to, the feature matrices of the (n-1)th and (n-2)th frames. After completing the depth prediction initialization, the systolic array 2 inputs the source image feature matrix and the reference image feature matrix into SRAM unit 2 for buffering.

[0068] Step 4: Before performing the current depth prediction operation, the general-purpose processor determines whether the previous depth prediction operation has ended. If it has, it sends an instruction to the systolic array 2 and the auxiliary computing module to perform the current depth prediction operation; if it has not ended, it waits for the previous depth prediction operation to end before sending an instruction to the systolic array 2 and the auxiliary computing module to perform the current depth prediction operation.

[0069] Step 5: The systolic array 2 and the auxiliary computation module perform depth prediction calculations. Before each step of the depth prediction calculation, the general-purpose processor performs algorithm analysis. If the calculation can be decomposed into a multiplication-accumulation operation, it is input into the systolic array 2 for calculation; if the calculation cannot be decomposed into a multiplication-accumulation operation, it is input into the auxiliary computation module for calculation. During the depth prediction process, the general-purpose processor sends information such as the source image feature matrix, reference image feature matrix, or Rt matrix to the systolic array 2 or the auxiliary computation module for depth prediction based on the calculation requirements of each step. After the calculation is completed, the general-purpose processor sends a signal to the data management module indicating that the depth prediction operation is complete, and caches the calculated depth information and confidence information into SRAM unit 2. After receiving the depth prediction completion signal, the data management module copies the predicted depth information to the secondary cache.

[0070] Part 2 is complete.

[0071] Part 3, Deep Integration:

[0072] In step 1, the data management module copies the RGB matrix information of the (n-1)th frame image, the RGB matrix information of the nth frame image, the predicted depth information, the confidence information, and the Rt matrix information from the secondary cache to the SRAM unit 3, and waits for the pulsating array 3 to complete the previous depth fusion operation.

[0073] Step 2: Before performing the deep fusion operation, the general-purpose processor first determines whether the previous deep fusion operation has ended. If it has ended, it sends an instruction to the systolic array 3 and the auxiliary computing module to perform the current deep fusion operation; if it has not ended, it waits for the previous deep fusion operation to end before sending an instruction to the systolic array 3 and the auxiliary computing module to perform the current deep fusion operation.

[0074] Step 3: The pulsating array 3 and the auxiliary computing module perform deep fusion operations. Before each step of the deep fusion operation, the general-purpose processor performs algorithm analysis. If the next operation can be decomposed into multiplication and accumulation operations, it is input into the pulsating array 3 for operation; if the next operation cannot be decomposed into multiplication and accumulation operations, it is input into the auxiliary computing module for operation. During the deep fusion operation, the general-purpose processor will send information such as the RGB matrix of the (n-1)th frame image, the RGB matrix of the nth frame image, depth, confidence, or Rt matrix to the pulsating array 2 or the auxiliary computing module for deep fusion operation according to the calculation needs of each step. At the same time, the pulsating array 3 and the general-purpose processor perform loop closure detection based on the RGB matrix of the (n-1)th frame image and the RGB matrix of the nth frame image, and use the loop closure detection results to guide the deep fusion process (if no loop closure is found, deep fusion is performed normally; if a loop closure is found, loop closure correction is performed). After the operation is completed, the general-purpose processor sends a signal to the data management module that the deep fusion operation is complete, and caches the 3D point cloud information into SRAM unit 3. After receiving the deep fusion completion signal, the data management module inputs the 3D point cloud information into the main memory for storage, awaiting subsequent access by peripheral screens. Simultaneously, the data management module deletes all information from the (n-2)th frame image in the secondary buffer and sends itself a signal indicating the completion of this 3D reconstruction (operations include, but are not limited to, setting a register bit to 1 or 0).

[0075] It should be noted that the time required for operations on different parts during the 3D reconstruction process varies, such as... Figure 5 As shown, the time required for different operations is illustrated. It can be seen that feature point matching is slightly faster than depth prediction. Therefore, to better implement pipelined processing, multiple depth prediction units can be configured, i.e., multiple matrix units can be configured to perform depth prediction operations. A selector determines which depth prediction unit is idle and transfers data to that idle unit. If none are idle, the data is temporarily stored in a data buffer unit. The DMA controller unit is mainly used for data transfer during peak data processing periods or for large-scale data transfer. When the general-purpose processor and register unit are operating at peak capacity, data transfer is performed through the DMA controller unit. For large-scale data transfers, such as RGB feature matrices of consecutive frames of images, using the general-purpose processor and data management module for data transfer is inefficient; in this case, the DMA controller unit is used for large-scale data transfer.

[0076] like Figure 6 The diagram shown illustrates the computational load allocation of an accelerator system according to one embodiment, reflecting the computational load allocation and data flow within the system. The hardware units responsible for the computational load and its allocation include:

[0077] Pulsating array 1 is used to perform feature point matching operations to obtain matching pixel information;

[0078] Pulsating array 2 is used for depth prediction to obtain image depth information and confidence information;

[0079] Pulsating array 3 is used for deep fusion to obtain three-dimensional point cloud information;

[0080] A general-purpose processor is used to perform polynomial calculations to obtain the camera's extrinsic rotation matrix Rt.

[0081] The auxiliary calculation module is used to perform calculations that are difficult to decompose in 3D reconstruction.

[0082] Data flow includes:

[0083] Data flow direction 1: Continuous frame images are input to pulse array 1, pulse array 2 and pulse array 3 respectively;

[0084] Data flow direction 2: Matched pixels and camera correction information are input to the general processor;

[0085] Data flow direction 3: The Rt rotation matrix is ​​input to pulsation array 2 and pulsation array 3;

[0086] Data flow direction 4: Data exchange between the auxiliary computing module and pulsating arrays 2 and 3 for bidirectional computing;

[0087] Data flow direction 5: Image depth information and confidence information are input to matrix operation unit 3;

[0088] Data flow direction 6: 3D point cloud information is output to peripheral devices.

[0089] like Figure 6 The diagram shown is a time flow graph of a three-stage pipeline for a 3D reconstruction accelerator according to one embodiment, reflecting the specific hardware resource allocation and runtime distribution during the implementation of the three-stage pipeline.

[0090] To ensure the smooth operation of the 3D reconstruction algorithm, hardware resources need to be allocated reasonably according to the distribution of the accelerator's 3D reconstruction computational load. In this embodiment, it is assumed that feature point matching and Rt rotation matrix calculations each occupy one runtime cycle; depth prediction and depth fusion each occupy two runtime cycles.

[0091] The hardware resource allocation ratio at this time includes: x feature matching operation units, x Rt rotation matrix operation units, 2x depth prediction operation units, and 2x depth fusion operation units.

[0092] The data flow is as follows:

[0093] First, feature point matching is performed. Then, Rt rotation matrix operations are performed. Next, a depth prediction unit is selected for computation based on a gating mechanism; the gating mechanism determines which depth prediction units are idle and transmits data to the first idle depth prediction unit for computation. Finally, a depth fusion unit is selected for computation based on a gating mechanism; the gating mechanism determines which depth fusion units are idle and transmits data to the first idle depth fusion unit for computation.

[0094] Compared with existing technologies, the advantages of this control method are as follows: This invention distributes the three computationally intensive operations—feature point matching, depth prediction, and depth fusion—to three matrix operation units. By rationally allocating the computation time and operator ratio of each matrix operation unit, the operations between the units can be made independent of each other, achieving the parallel processing effect of a three-stage pipeline. This parallel processing architecture improves the efficiency and compatibility of the accelerator for image processing in 3D reconstruction tasks.

[0095] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0096] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0097] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0098] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A neural network accelerator device for image processing, characterized in that, The device includes: The multiply-accumulate operation module is configured with multiple matrix operation units for continuous accumulation and multiplication operations. Specifically, multiple matrix operation units within the multiply-accumulate operation module are configured to handle neural network operations for feature point matching, depth prediction, and depth fusion. The allocation of resources among these units is based on their computation time and operator ratios to ensure concurrent feature point matching, depth prediction, and depth fusion operations, forming a three-stage pipelined operation. The data storage module is configured as a multi-level storage structure for hierarchical multi-level storage of data and for data sharing. The data management module is used to retrieve computational data and allocate storage space. A general-purpose processor is used to perform the calculation of the extrinsic parameter Rt rotation matrix, which is used for depth prediction and depth fusion calculations; The auxiliary operation module is used to perform operations that cannot be decomposed, except for multiplication and accumulation operations and solving the external parameter Rt rotation matrix.

2. The apparatus according to claim 1, characterized in that, The multiply-accumulate operation module includes three matrix operation units, each of which is configured with a corresponding data cache unit. Each matrix operation unit is used to perform multiplication operations, and each data cache unit is used to directly exchange data with its corresponding matrix operation unit.

3. The apparatus according to claim 1, characterized in that, Each matrix operation unit includes multiple processing units and multiple array registers. Each processing unit includes at least one multiplier and one adder. The multiple array registers are used to transfer intermediate data processed by the processing unit.

4. The apparatus according to claim 1, characterized in that, The multiply-accumulate operation module, general-purpose processor, and auxiliary operation module can be configured in a pipelined mode.

5. The apparatus according to claim 1, characterized in that, The data storage module includes a secondary cache and main memory, wherein: The secondary cache is directly connected to each data cache unit in the multiply-accumulate operation module and is used to store data that cannot be stored in each data cache unit. The main memory is directly connected to the secondary cache and is used to store large amounts of data that cannot be stored in the secondary cache.

6. The apparatus according to claim 1, characterized in that, The data management module includes a register unit, a selector unit, a DMA controller unit, and a data temporary storage unit, wherein: The register unit is used for transferring intermediate data; The selector unit is used to select multiple matrix operation units to achieve multi-stage pipelined operation; The DMA controller unit is used to arrange and transport image data in the DMA path. The data temporary storage unit is used to temporarily store data during peak data transmission periods.

7. The apparatus according to claim 1, characterized in that, The general-purpose processor is configured to solve polynomials, solve Jacobian matrices, and calculate the extrinsic Rt rotation matrix at adjacent frame times of an image.

8. The apparatus according to claim 1, characterized in that, The auxiliary calculation module includes: The logic operation unit is used to perform various logic operations, including at least AND, OR, NOT, XOR, comparison, AND / OR, and shift logic operations. Arithmetic operation unit, used to perform operations on small-scale data that cannot be decomposed, including at least addition, subtraction, multiplication, division, squaring, sum / or, and square root arithmetic operations; The activation function operation unit is used to perform activation operations on data, including at least the activation operations of the ReLU activation function, the binary activation function, and / or the Sigmoid activation function; The data sorting unit is used to call the data in the data management module to perform data sorting operations, including sorting by finding the maximum value, sorting by finding the minimum value, and / or sorting the data. A pooling operation unit is used to assist the matrix operation unit in performing pooling calculations in the neural network. An upsampling unit is used to assist the matrix operation unit in performing upsampling calculations in the neural network.

9. A three-dimensional reconstruction method based on the device described in any one of claims 1-8, characterized in that, The method includes configuring multiple matrix operation units in the multiply-accumulate operation module to be responsible for feature point matching, multiple matrix operation units to be responsible for depth prediction, and multiple matrix operation units to be responsible for depth fusion, and allocating them according to the operation time and operator ratio of each matrix operation unit so that feature point matching, depth prediction, and depth fusion operations are performed concurrently to form a three-level pipeline mode; after acquiring the RGB information of continuous frame images and camera correction information collected by the camera peripheral, the following steps are performed for each frame image: S1, the matrix operation unit, data management module and general processor responsible for feature point matching perform feature point matching operations based on image RGB information and camera correction information, and pass the feature point matching results to the general processor for calculation of the external parameter Rt rotation matrix; S2, the matrix operation unit, data management module and auxiliary operation module responsible for depth prediction perform depth prediction operation based on RGB image information and Rt rotation matrix, and pass the depth prediction result to the matrix operation unit responsible for depth fusion; S3, the matrix operation unit, data management module and auxiliary operation module responsible for deep fusion, perform deep fusion operations based on RGB image information, Rt rotation matrix and depth prediction results to obtain three-dimensional point cloud information.

10. The method according to claim 9, characterized in that, The matrix operation units are allocated according to the time cycles required for feature point matching, depth prediction, and deep fusion to ensure that the clock cycles of each node in the pipeline formed by feature point matching, depth prediction, and deep fusion are balanced.

11. The method according to claim 9, characterized in that, After obtaining the RGB information of the continuous frame images captured by the camera peripheral, the data management module calls it to store it in the main memory.

12. The method according to claim 11, characterized in that, After receiving the signal that the previous 3D reconstruction operation was completed, the data management module copies the RGB information of the current frame image from main memory to the secondary cache.

13. The method according to claim 12, characterized in that, Step S1 includes: S11. The data management module copies the RGB information of the current frame image from the secondary cache to the data cache unit corresponding to the matrix operation unit responsible for feature point matching. S12. After receiving the signal that the matrix operation unit responsible for feature point matching has completed the previous feature point matching, the RGB information of the current frame image and the previous frame image in the corresponding data buffer unit is transmitted to the matrix operation unit responsible for feature point matching, and the RGB information of the frame before the previous one stored in the data buffer unit is deleted. S13. The matrix operation unit responsible for feature point matching performs feature point matching based on the RGB information of the current frame image and the previous frame image in its corresponding data cache unit. After the feature point matching is completed, the matching pixel points are stored in the corresponding data cache unit and a feature point matching completion signal is sent to the data management module. S14. After receiving the feature point matching completion signal of the matrix operation unit responsible for feature point matching for the current frame image and the Rt rotation matrix calculation completion signal of the general processor, the data management module transmits the matching pixel points stored in the data cache unit and the camera correction information in the main memory to the general processor. S15. The general-purpose processor calculates the external parameter Rt rotation matrix based on the matching pixel points and camera correction information transmitted by the data management module, and sends an Rt rotation matrix calculation completion signal after the calculation is completed. S16. After receiving the signal that the Rt rotation matrix calculation is complete, the data management module stores the calculated external parameter Rt rotation matrix information of the camera for the two consecutive frames into the secondary buffer.

14. The method according to claim 13, characterized in that, Step S2 includes: S21. The data management module copies the RGB information and Rt rotation matrix information of the current frame image from the secondary buffer to the data buffer unit corresponding to the matrix operation unit responsible for depth prediction. S22. After receiving the signal that the matrix operation unit responsible for depth prediction has completed the previous depth prediction, the data management module transmits the RGB information of the current frame image and one or more historical frame images in the corresponding data cache unit to the matrix operation unit responsible for depth prediction and deletes the RGB information of unused images stored in the data cache unit. S23. The matrix operation unit responsible for depth prediction performs depth prediction based on the RGB information of the current frame image, the Rt rotation matrix information, and the RGB information of one or more historical frame images in its corresponding data cache unit. After the prediction is completed, it sends a depth prediction completion signal to the data management module. S24. After receiving the depth prediction completion signal, the data management module stores the depth information and confidence information obtained from the depth prediction into the secondary cache.

15. The method according to claim 14, characterized in that, Step S3 includes: S31. The data management module copies the RGB matrix information of the previous frame image, the RGB matrix information of the current frame image, the predicted depth information, the confidence information, and the Rt rotation matrix from the secondary cache to the data cache unit corresponding to the matrix operation unit responsible for depth fusion. S32. After receiving the signal that the previous deep fusion operation is completed, the data management module transmits the RGB matrix information of the previous frame image, the RGB matrix information of the current frame image, the predicted depth information, the confidence information, and the Rt rotation matrix in the corresponding cache unit to the matrix operation unit responsible for deep fusion. S33. The matrix operation unit responsible for deep fusion performs deep fusion on the previous frame image RGB matrix information, the current frame image RGB matrix information, the predicted depth information, the confidence information and the Rt rotation matrix in the corresponding buffer unit after the signal of the completion of the previous deep fusion operation. After the fusion is completed, it sends a deep fusion completion signal to the data management module. S34. After receiving the deep fusion completion signal, the data management module stores the 3D point cloud information obtained by deep fusion into the main memory.

16. A computer-readable storage medium, characterized in that, It contains a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 9 to 15.

17. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the electronic device to perform the steps of the method as described in any one of claims 9 to 15.

Images