Configurable depth-aware convolution operator hardware architecture

Abstract

This disclosure provides a configurable depth-aware convolution operator hardware architecture applicable to the field of hardware design technology. It includes: a task scheduling module for generating data enable signals and controlling the execution order of convolution tasks based on these signals; a configurable module for determining the operating mode based on a mode enable signal; a depth value caching module for determining at least one depth value corresponding to the operating mode upon receiving a depth value determination enable signal, and for sending at least one depth value corresponding to the operating mode to a depth coefficient calculation module upon receiving a depth value transmission enable signal; a depth coefficient calculation module for obtaining a set of depth coefficients corresponding to at least one depth value; a pre-computation module for pre-computing the depth coefficient set and the input feature map to obtain a pre-computation result; and a systolic array for performing convolution processing on the pre-computation result to obtain a convolution processing result.

Description

Technical Field

[0001] This disclosure relates to the field of hardware design technology, and more specifically to a hardware architecture for a configurable depth-aware convolution operator. Background Technology

[0002] In recent years, Convolutional Neural Networks (CNNs) have been widely used in computer vision, especially in tasks such as image classification and object detection. CNNs extract spatial features from images through two-dimensional convolution operations, making them suitable for most scenarios. However, with the development of depth sensors, the combination of depth images and RGB images can provide more geometric information, improving the accuracy of scene understanding. Traditional convolutional neural networks, due to their fixed mesh structure, struggle to effectively handle geometric variability in images, limiting their utilization of depth information.

[0003] To address this issue, researchers proposed a depth-aware convolution operator. By introducing a depth similarity term into traditional convolutional neural networks, pixels with similar depths have a greater impact on the convolution result. Compared to two-stream networks or 3D convolution, depth-aware convolution can flexibly integrate geometric information and improve the understanding of scene geometry without significantly increasing computational complexity. However, the inventors found during their research that the hardware architectures in related technologies are mostly based on traditional convolutional neural network designs, which cannot fully leverage the advantages of the depth-aware convolution operator and reduce its application efficiency. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a configurable depth-aware convolution operator hardware architecture.

[0005] According to one aspect of this disclosure, a configurable depth-aware convolution operator hardware architecture is provided, comprising:

[0006] The task scheduling module is used to generate data enable signals and control the execution order of convolution tasks based on the data enable signals. The data enable signals include mode enable signals and depth value transmission enable signals.

[0007] A configurable module is used to determine the working mode based on the mode enable signal and send the depth value determination enable signal corresponding to the working mode to the depth value cache module. The working mode includes a regular convolution mode or a depth-aware convolution mode.

[0008] The depth value caching module is used to determine at least one depth value corresponding to the working mode when a depth value determination enable signal is received, and to send at least one depth value corresponding to the working mode to the depth coefficient calculation module when a depth value transmission enable signal is received.

[0009] A depth coefficient calculation module, configured to obtain depth coefficients corresponding to at least one depth value based on at least one depth value, and send the depth coefficients corresponding to the at least one depth value to a pre-calculation module;

[0010] A pre-calculation module, configured to perform pre-calculation on the depth coefficients corresponding to the at least one depth value and an input feature map to obtain a pre-calculation result, and send the pre-calculation result to a systolic array;

[0011] A systolic array, configured to perform convolution processing on the pre-calculation result to obtain a convolution processing result.

[0012] According to an embodiment of the present disclosure, the depth coefficient calculation module includes: an absolute value calculation matrix, at least one data stacking unit, and at least one absolute value caching unit;

[0013] The absolute value calculation matrix is configured to determine a reference depth value from the at least one depth value, and determine the absolute value difference between the reference depth value and each of the at least one depth value;

[0014] Each of the at least one data stacking unit corresponds to at least one target absolute value calculation unit in the absolute value calculation matrix. The data stacking unit is configured to splice the absolute value differences output by the at least one target absolute value calculation unit in the same system clock cycle to obtain stacked data, where the at least one target absolute value calculation unit is located in the same row;

[0015] Each of the at least one absolute value caching unit is respectively connected to the at least one data stacking unit. The absolute value caching unit is configured to cache the stacked data.

[0016] According to an embodiment of the present disclosure, the absolute value calculation matrix includes J columns of absolute value calculation units;

[0017] When in the i-th system clock cycle and i is not equal to N*J,

[0018] In response to j = 1, the absolute value calculation unit in the j-th column of the absolute value calculation matrix is configured to load the target depth value from at least one depth value sent by the depth value caching module, and send the target depth value to the absolute value calculation unit in the j+1-th column, where i is an integer greater than or equal to 1, and N and J are integers;

[0019] In response to 1 < j < J, the absolute value calculation unit in the j-th column is configured to load the target depth value sent by the absolute value calculation unit in the j-1-th column, and send the target depth value to the absolute value calculation unit in the j+1-th column;

[0020] In response to j = J, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the absolute value calculation unit of the (j - 1)-th column;

[0021] In the case where it is the i-th system clock cycle and i is equal to N * J,

[0022] In response to j = 1, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the depth value cache module, and is configured to determine the target absolute value difference between the target depth value and the reference depth value, and is configured to send the target depth value to the absolute value calculation unit of the (j + 1)-th column, and send the target absolute value difference to the data stacking unit corresponding to the absolute value unit, wherein, the reference depth value is the M-th target depth value sent by the depth value cache module, and M is an integer greater than or equal to 1;

[0023] In response to 1 < j < J, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the absolute value calculation unit of the (j - 1)-th column, and is configured to determine the target absolute value difference between the target depth value and the reference depth value, and is configured to send the target depth value to the absolute value calculation unit of the (j + 1)-th column, and send the target absolute value difference to the data stacking unit corresponding to the absolute value unit;

[0024] In response to j = J, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the absolute value calculation unit of the (j - 1)-th column, and is configured to determine the target absolute value difference between the target depth value and the reference depth value, and is configured to send the target absolute value difference to the data stacking unit corresponding to the absolute value unit, and delete the target depth value.

[0025] According to an embodiment of the present disclosure, the absolute value cache unit includes a first selector, at least two cache subunits, and a second selector;

[0026] The first selector is configured to determine a first cache subunit from at least two cache subunits, wherein the first cache subunit is configured to cache the written stacked data;

[0027] The second selector is configured to determine a second cache subunit from at least two absolute value cache units, and the second cache subunit is configured to provide the stacked data for the asynchronous buffer unit to read the stacked data.

[0028] According to an embodiment of the present disclosure, the depth coefficient calculation module further includes: an asynchronous buffer unit.

[0029] The asynchronous buffer unit is configured to obtain at least one target stacked data from the absolute value cache unit corresponding to the asynchronous buffer unit, and temporarily store the at least one target stacked data according to the storage capacity threshold of the asynchronous buffer unit.

[0030] According to embodiments of this disclosure, the input feature map includes at least one input feature value; the pre-computation module includes a lookup unit and a pre-computation unit;

[0031] The lookup unit is used to obtain the target absolute value difference from at least one target stack data stored in the asynchronous buffer unit based on a preset data volume, and to determine the depth coefficient corresponding to the target absolute value difference from a preset lookup table based on the target absolute value difference.

[0032] The pre-computation unit is used to multiply the depth coefficient with the corresponding input feature value to obtain the pre-computation sub-result corresponding to the depth coefficient, and to obtain the pre-computation result with the pre-computation sub-result corresponding to at least one depth coefficient, and to send the pre-computation result to the pulsation array.

[0033] According to embodiments of this disclosure, the depth value caching module includes a third selector and a depth value caching unit;

[0034] The depth value cache unit is used to cache at least one depth value;

[0035] When the working mode is regular convolution mode, the third selector is used to output a depth value equal to the preset value;

[0036] When the working mode is depth-aware convolution mode, the third selector is used to output at least one depth value from the depth value cache unit in sequence.

[0037] According to embodiments of this disclosure, the configurable depth-aware convolution operator hardware architecture further includes: an input feature map caching module and a weight caching module;

[0038] The input feature map caching module is used to cache the input feature map and send the input feature map to the pre-computation module when the input data transmission enable signal sent by the task scheduling module is received.

[0039] The weight caching module is used to cache at least one weight value and, upon receiving a weight transmission enable signal from the task scheduling module, to send at least one weight value to the systolic array so that the systolic array can perform convolution processing on the pre-calculated result based on at least one weight value to obtain the convolution processing result.

[0040] According to embodiments of this disclosure, the configurable depth-aware convolutional operator hardware architecture further includes a cached data loading module; the cached data loading module includes an input feature map loading unit, a depth value loading unit, and a weight loading unit;

[0041] The input feature map loading unit is used to load at least one depth value into the depth value cache module when it receives an input feature map loading enable signal sent by the task scheduling module.

[0042] The depth value loading unit is used to load the input feature map into the input feature map cache module when the depth value loading enable signal is received from the task scheduling module.

[0043] The weight loading unit is used to load at least one weight value into the weight cache module when it receives a weight loading enable signal sent by the task scheduling module.

[0044] According to embodiments of this disclosure, the configurable depth-aware convolution operator hardware architecture further includes: a post-processing module;

[0045] The post-processing module is used to perform pooling on the convolution results to obtain pooled data, and then to map the pooled data to obtain the target output data.

[0046] According to the configurable depth-aware convolution operator hardware architecture disclosed herein, the execution order of convolution tasks is controlled by a task scheduling module that generates a data enable signal. For example, if a mode enable signal is generated, the configurable module determines the working mode through the mode enable signal and controls the depth value caching module to output at least one depth value that satisfies the working mode. This ensures that the depth coefficients calculated using the at least one depth value meet the computational requirements of the working mode. The pre-computation module pre-computes the depth coefficients and the input feature map and sends the pre-computation results to the systolic array for convolution processing. This enables the systolic array to be used normally in different working modes, thereby at least partially solving the technical problem that the hardware architecture in related technologies cannot fully utilize the advantages of the depth-aware convolution operator. It enables the hardware system to flexibly switch between the conventional convolution mode and the depth-aware convolution mode, better leveraging the advantages of the depth-aware convolution operator. Furthermore, in the implementation process, the data flow of the conventional convolution mode and the depth-aware convolution mode is basically the same. While being compatible with different working modes, it can share the data flow and various modules, reducing the complexity of data flow scheduling and saving certain hardware resources. Attached Figure Description

[0047] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0048] Figure 1 A schematic diagram of a configurable depth-aware convolution operator hardware architecture according to a first embodiment of the present disclosure is shown.

[0049] Figure 2 A schematic diagram illustrating a configurable depth-aware convolution operator hardware architecture according to a second embodiment of the present disclosure is shown.

[0050] Figure 3A schematic diagram illustrating the architecture of the depth coefficient calculation module and the pre-calculation module according to a second embodiment of the present disclosure is shown.

[0051] Figure 4 A schematic diagram illustrating the task processing flowchart of a configurable depth-aware convolution operator hardware architecture according to an embodiment of the present disclosure is provided. Detailed Implementation

[0052] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0053] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0054] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0055] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0056] Embodiments of this disclosure provide a configurable depth-aware convolutional operator hardware architecture, characterized by comprising: a task scheduling module, used to generate data enable signals and control the execution order of convolutional tasks based on the data enable signals, wherein the data enable signals include mode enable signals and depth value transmission enable signals; a configurable module, used to determine a working mode based on the mode enable signals and send a depth value determination enable signal corresponding to the working mode to a depth value cache module, wherein the working mode includes a regular convolution mode or a depth-aware convolution mode; a depth value cache module, upon receiving a depth value determination enable signal, determining at least one depth value corresponding to the working mode, and upon receiving a depth value transmission enable signal, sending at least one depth value corresponding to the working mode to a depth coefficient calculation module; a depth coefficient calculation module, used to obtain a set of depth coefficients corresponding to at least one depth value based on at least one depth value, and sending the depth coefficient set to a pre-calculation module; a pre-calculation module, used to pre-calculate the depth coefficient set and the input feature map to obtain a pre-calculation result, and sending the pre-calculation result to a systolic array; and a systolic array, used to perform convolution processing on the pre-calculation result to obtain a convolution processing result.

[0057] Figure 1 A schematic diagram of a configurable depth-aware convolution operator hardware architecture according to a first embodiment of the present disclosure is shown.

[0058] like Figure 1 As shown, the configurable depth-aware convolution operator hardware architecture includes the following modules.

[0059] The task scheduling module 101 is used to generate data enable signals and control the execution order of convolution tasks based on the data enable signals. The data enable signals include mode enable signals and depth value transmission enable signals.

[0060] The configurable module 102 is used to determine the working mode based on the mode enable signal and send the depth value determination enable signal corresponding to the working mode to the depth value cache module 103. The working mode includes a regular convolution mode or a depth-aware convolution mode.

[0061] The depth value caching module 103 is used to determine at least one depth value corresponding to the working mode when a depth value determination enable signal is received, and to send at least one depth value corresponding to the working mode to the depth coefficient calculation module 104 when a depth value transmission enable signal is received.

[0062] The depth coefficient calculation module 104 is used to obtain the depth coefficient corresponding to the at least one depth value based on at least one depth value, and send the depth coefficient corresponding to the at least one depth value to the pre-calculation module 105.

[0063] The pre-calculation module 105 is used to pre-calculate the depth coefficient corresponding to at least one depth value and the input feature map to obtain the pre-calculation result, and send the pre-calculation result to the pulsation array 106.

[0064] The pulsating array 106 is used to perform convolution processing on the pre-calculated results to obtain the convolution processing result.

[0065] According to embodiments of this disclosure, Figure 1 The dashed lines in the diagram represent the direction of signal transmission, while the solid lines represent the direction of data transmission.

[0066] According to an embodiment of this disclosure, the task scheduling module 101 can determine the current working mode based on the received working mode signal, and send a mode enable signal corresponding to the working mode to the configurable module 102.

[0067] According to embodiments of this disclosure, the task scheduling module 101 is used to coordinate the task execution order of each module to ensure that data processing and computation can be performed sequentially. Specifically, the task scheduling module 101 is used to generate a scheduling plan based on task priority and availability of computing resources to ensure that convolution computation tasks are performed in sequence. The generation order and timing of each data enable signal are not limited, and different generation order and timing of data enable signals can be determined according to requirements.

[0068] According to embodiments of this disclosure, the task scheduling module 101 generates a data enable signal to ensure that data is loaded from a designated module to another module at a specific time. For example, the task scheduling module 101 generates an enable signal for loading input feature map, weights, and depth map data. In depth-aware convolution mode: the input feature map and depth values ​​are first scheduled for pre-calculation of depth-aware coefficients, and then the weight values ​​are scheduled to the systolic array 106. In ordinary perceptual convolution mode, since the depth value caching module 103 outputs preset values ​​for subsequent calculations, depth value transmission does not require bus bandwidth; only the input feature map data and weight data need to be scheduled and deployed to the systolic array 106 for ordinary convolution operations. When data enters the depth coefficient calculation module 104, the calculation enable signal needs to be enabled only after the data has completely filled the calculation array.

[0069] According to embodiments of this disclosure, the configurable module 102 can be used to determine the current working mode based on the mode enable signal, and to control the registers included in the configurable module 102 to output 1 or 0 to characterize the current working mode. For example, mode=1 can represent working in depth-aware convolution mode, and mode=0 indicates working in regular convolution mode.

[0070] According to embodiments of this disclosure, the configurable mode module can control the data flow direction through configuration registers according to system requirements, thereby enabling switching between ordinary convolution and depth-aware convolution modes.

[0071] According to embodiments of this disclosure, at least one depth value is used to construct a depth map, the depth map including at least one depth value.

[0072] According to embodiments of this disclosure, the depth value cache module 103 can be used to determine the output command indicated by the enable signal based on the depth value, and output at least one depth value corresponding to the working mode. For example, when the working mode is depth-aware convolution mode, the depth value of the depth map existing in the depth value cache module 103 is output, and when the working mode is depth-aware convolution mode, at least one preset value is output.

[0073] According to embodiments of this disclosure, the depth coefficient calculation module 104 is used to dynamically calculate the depth coefficients required during the convolution process based on at least one depth value.

[0074] According to an embodiment of this disclosure, the input feature map pre-computation module 105 is used to preprocess the input feature values ​​and depth coefficients included in the input feature map.

[0075] According to embodiments of this disclosure, the systolic array 106, through multiple parallel processing units (PEs), works collaboratively to achieve parallel convolution computation of input feature maps, depth maps, and weight data. Each processing unit is responsible for a specific computational task and ultimately outputs the convolution result. PEs can be computing devices such as processors, computers, and servers.

[0076] According to embodiments of this disclosure, the depth coefficient calculation module 104 performs dynamic depth coefficient calculation based on the depth map and transmits the calculated depth coefficients to the input pre-calculation module 105. The pre-calculation module 105 performs preliminary processing on the input feature map and depth coefficients to provide preprocessing results for the convolution operation of the systolic array 106.

[0077] According to embodiments of this disclosure, the execution order of convolution tasks is controlled by a data enable signal generated by a task scheduling module 101. For example, a mode enable signal is generated, and a configurable module 102 determines the working mode through the mode enable signal and controls the depth value cache module 103 to output at least one depth value that satisfies the working mode. This ensures that the depth coefficients calculated using the at least one depth value meet the calculation requirements of the working mode. The pre-calculation module 105 pre-calculates the depth coefficients and the input feature map and sends the pre-calculation results to the systolic array 106 for convolution processing. This enables the systolic array 106 to be used normally in different working modes, thereby at least partially solving the technical problem that the hardware architecture in related technologies cannot fully utilize the advantages of depth-aware convolution operators. It enables the hardware system to flexibly switch between conventional convolution mode and depth-aware convolution mode, better leveraging the advantages of depth-aware convolution operators. Furthermore, in the implementation process, the data flow of conventional convolution mode and depth-aware convolution is basically the same. While being compatible with different working modes, it can share data flow and various modules, reducing the complexity of data flow scheduling and saving certain hardware resources.

[0078] Figure 2 A schematic diagram of a configurable depth-aware convolution operator hardware architecture according to a second embodiment of the present disclosure is shown.

[0079] like Figure 2 As shown, the configurable depth-aware convolutional operator hardware architecture includes the following modules: external memory 111, external memory controller 112, cached data loading module 107, input feature map cache module 108, depth value cache module 103, weight value cache module 109, task scheduling module 101, configurable module 102, depth coefficient calculation module 104, pre-computation module 105, systolic array 106, and post-module 110.

[0080] According to embodiments of this disclosure, Figure 2 The dashed lines in the diagram can represent the direction of control signal flow, while the solid lines can represent the direction of data flow.

[0081] According to embodiments of this disclosure, the configurable module 102 can configure the registers of each module to enable the configurable depth-aware convolution operator hardware architecture to operate in either a conventional convolution mode or a depth-aware convolution mode. Mode switching is achieved by switching the data stream using a data selector.

[0082] According to embodiments of this disclosure, the hardware system can flexibly switch between conventional convolution mode and depth-aware convolution mode. This architecture can rationally configure computing resources and optimize computing performance in different application scenarios to meet diverse computing needs. Furthermore, through the parallel computing characteristics and configurable mode modules of the systolic array 106, it fully utilizes the computing units and on-chip cache of the systolic array 106 to achieve efficient data processing and convolution operations.

[0083] Figure 3 The schematic diagram illustrates the architecture of the depth coefficient calculation module 104 and the pre-calculation module 105 according to a second embodiment of the present disclosure.

[0084] like Figure 3 As shown, the depth value caching module 103 may include a third selector, and the depth coefficient calculation module 104 includes: an absolute value calculation matrix, at least one data stacking unit, at least one absolute value caching unit, and an asynchronous buffer unit; the pre-calculation module 105 includes a lookup unit and a pre-calculation unit.

[0085] According to embodiments of this disclosure, an absolute value calculation matrix is ​​used to calculate the depth difference of the input data, and a data stacking unit concatenates the results output by the absolute value calculation matrix. The concatenated data is stored in an absolute value cache unit including RAM (Random Access Memory), and then quickly queried and processed through a lookup unit including ROM (Read Only Memory). The entire module is coordinated by a clock signal to ensure accurate data processing and efficient transmission.

[0086] According to embodiments of this disclosure, taking a 3x3 depth-aware convolution operator as an example, a 3x3 absolute value calculation matrix is ​​required. This adaptively necessitates three third selectors, three data stacking units, three absolute value buffering units, three asynchronous buffering units, three lookup units, and three pre-computation units.

[0087] According to embodiments of this disclosure, at least one third selector can select an output depth value dp_1, dp_2, dp_3 or a preset value 1'b1 through a working mode mode.

[0088] According to embodiments of the present disclosure, the depth map includes the depth value of each pixel, that is, a depth value corresponds to a pixel. In depth-aware convolution, the depth similarity between each pixel is defined by the following formula (1).

[0089] (1)

[0090] in, It is a constant. and Representing pixels and The depth value.

[0091] According to an embodiment of this disclosure, in the first system clock cycle of activating the depth coefficient calculation module 104, the depth values ​​dp_1, dp_2, and dp_3 read from the depth value cache module 103 are input into the register reg of the first-level calculation array, specifically the register reg of the absolute value calculation unit located in the first column. Each cycle, data flows to the right along the horizontal direction of the absolute value calculation matrix and is loaded once. The calculation array is filled in the third system clock cycle. Wherein, according to... Figure 3 As can be seen from the reg loading status, the depth value in the first row and third column can be the base depth value D(p) i This is used by the other calculation units to calculate the absolute value difference.

[0092] According to embodiments of this disclosure, cal is a calculator in the absolute value calculation unit, and the calculator can be used to determine the relationship between any depth value and D(p). i The absolute value of the depth is used to determine the relationship between the depth values, and the difference between the depth values ​​is obtained by subtracting the depth value with the smaller absolute value from the depth value with the larger absolute value.

[0093] According to embodiments of this disclosure, since combinational logic is used to calculate the absolute value of the depth difference, the absolute value difference corresponding to the current pixel will be stored in the output register out_reg within the same system clock cycle. In order for the depth coefficient module to complete the calculation of the absolute value difference of one pixel in one system clock cycle, the results of all current output registers must be stored in RAM within one cycle. Therefore, a data stacking unit is introduced to concatenate the absolute value differences of the same row into stacked data with a width of 3 times and write it into the absolute value cache unit.

[0094] According to embodiments of this disclosure, the data stacking unit mainly includes at least one RAM and a first selector and a second selector. It adopts a ping-pong buffer structure, with one line of RAM caching one line of stacked data. The RAM can be represented as RAMk_n, where k represents the kth data stacking unit and n represents the nth RAM in the first data stacking unit.

[0095] According to the embodiments of this disclosure, since a depth coefficient lookup table is required subsequently, one column of valid data is required to be transmitted per system clock cycle, and the calculation is required to be performed at 3 times the original clock frequency. Therefore, an asynchronous buffer unit is introduced. The asynchronous buffer unit can be an asynchronous FIFO (First Input First Output) buffer. The storage capacity threshold is not limited and can be selected according to the actual situation, such as a depth of 16.

[0096] According to embodiments of this disclosure, by placing an asynchronous buffer unit after the absolute value buffer unit, data loss due to data being written too quickly can be avoided.

[0097] According to embodiments of this disclosure, the lookup unit may include a storage space, which may be a ROM. The ROM may store a preset lookup table, which includes the correspondence between the absolute value difference and the depth coefficient. The absolute value difference of the depth coefficient of the corresponding bit may be extracted each time and sent to the ROM for table lookup. The ROM stores the quantized value of the e-exponent and outputs it to the next-level pre-calculation unit.

[0098] According to embodiments of this disclosure, placing ROM after RAM offers an advantage while meeting computational requirements: reduced ROM resource usage. Specifically, if a lookup table is used before RAM, a 3x3 absolute value calculation matrix requires 9 ROM units; using a lookup table after RAM utilizes only one-third of the ROM, thus optimizing on-chip resources.

[0099] According to embodiments of this disclosure, the specific data transmission process between the depth coefficient calculation module 104 and the pre-calculation module 105 can be as follows: taking the first row of absolute value calculation units in the absolute value calculation matrix as an example, the third selector in the first row determines the enable signal to select the output depth value based on the depth value. The depth value includes at least one depth value that can characterize the depth map stored in the depth value cache module 103, and at least one preset value. In the conventional convolution mode, the third selector outputs the preset value, and in the depth-aware convolution mode, the third selector outputs the depth value.

[0100] According to an embodiment of this disclosure, taking the depth-aware convolution mode as an example, the third selector located in the first row outputs a first depth value to the absolute value calculation unit located in the first row and first column of the absolute value calculation matrix during the first system clock cycle when the depth coefficient calculation module 104 is turned on. The absolute value calculation unit loads the depth value into reg (reg is a register) and sends the first depth value to the absolute value calculation unit in the second column of the first row. During the second system clock cycle when the depth coefficient calculation module 104 is turned on, the third selector located in the first row outputs a second depth value to the absolute value calculation unit in the first row and first column. The absolute value calculation unit repeats the operation of the first system clock cycle, and the absolute value calculation unit in the second column of the first row sends the first depth value to the absolute value calculation unit in the third column of the first row. In the third system clock cycle when the depth coefficient calculation module 104 is turned on, the third selector in the first row outputs the third depth value to the absolute value calculation unit in the first row and first column. At the same time, the cal in the absolute value calculation unit calculates the absolute difference between the third depth value and the reference depth value. The cal in the absolute value calculation unit in the first row and second column calculates the absolute difference between the second depth value and the reference depth value. The cal in the absolute value calculation unit in the first row and third column calculates the absolute difference between the first depth value and the reference depth value. After obtaining the above absolute differences, they are stored in the output register out_reg of each absolute value calculation unit.

[0101] According to an embodiment of this disclosure, during the third or fourth system clock cycle when the depth coefficient calculation module 104 is turned on, each absolute value calculation unit stores the absolute value difference in its out_reg into the data stacking unit located in the first row, and the data stacking unit concatenates the three absolute value differences in sequence.

[0102] According to an embodiment of this disclosure, the data stacking unit sends a cache selection signal to a first selector in the absolute value cache unit located in the first row. The first selector determines a first cache sub-unit from at least two cache sub-units based on the cache selection signal to cache the stacked data.

[0103] According to an embodiment of this disclosure, an asynchronous buffer unit located in the first row sends a data read signal to a second selector in an absolute value buffer unit located in the first row. The second selector determines a second buffer sub-unit from at least two buffer sub-units based on the data read signal, so that the asynchronous buffer unit reads stacked data from the second buffer sub-unit.

[0104] According to an embodiment of the present disclosure, the lookup unit located in the first row reads the absolute value difference from the asynchronous buffer unit, finds the depth coefficient from a preset lookup table, and repeats the above process until the depth coefficients corresponding to at least one depth value are found respectively. After obtaining each depth coefficient, the depth coefficient is passed to the pre-computation unit for pre-computation.

[0105] According to an embodiment of the present disclosure, Figure 3 The numbers of the third selector, the absolute value calculation matrix, the data stacking unit, the absolute value cache unit, the asynchronous buffer unit, the lookup unit, and the pre-computation unit in [[ ]] are illustrative and not limited thereto. Different numbers can be adopted according to requirements.

[0106] According to an embodiment of the present disclosure, the absolute value calculation matrix is used to determine a reference depth value from at least one depth value and determine the absolute value difference between the reference depth value and each of the at least one depth value. At least one data stacking unit, each data stacking unit corresponding to at least one target absolute value calculation unit in the absolute value calculation matrix, the data stacking unit is used to splice the absolute value differences output by the at least one target absolute value calculation unit in the same system clock cycle to obtain stacked data, wherein at least one target absolute value calculation unit is located in the same row. At least one absolute value cache unit, respectively connected to at least one data stacking unit, the absolute value cache unit is used to cache the stacked data.

[0107] According to an embodiment of the present disclosure, the absolute value calculation matrix includes J columns of absolute value calculation units. When in the i-th system clock cycle and i is not equal to N*J, in response to j = 1, the absolute value calculation unit located in the j-th column of the absolute value calculation matrix is used to load the target depth value from at least one depth value sent by the depth value cache module 103 and send the target depth value to the absolute value calculation unit in the j + 1-th column, where i is an integer greater than or equal to 1, and N and J are integers; in response to 1 < j < J, the absolute value calculation unit in the j-th column is used to load the target depth value sent by the absolute value calculation unit in the j - 1-th column and send the target depth value to the absolute value calculation unit in the j + 1-th column; in response to j = J, the absolute value calculation unit in the j-th column is used to load the target depth value sent by the absolute value calculation unit in the j - 1-th column.

[0108] According to an embodiment of the present disclosure, in the case where it is the i-th system clock cycle and i is equal to N*J, in response to j = 1, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the depth value cache module 103, and is configured to determine the target absolute value difference between the target depth value and the reference depth value, and is configured to send the target depth value to the absolute value calculation unit of the j + 1-th column, and send the target absolute value difference to the data stacking unit corresponding to the absolute value unit, where the reference depth value is the M-th target depth value sent by the depth value cache module 103, and M is an integer greater than or equal to 1; in response to 1 < j < J, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the absolute value calculation unit of the j - 1-th column, and is configured to determine the target absolute value difference between the target depth value and the reference depth value, and is configured to send the target depth value to the absolute value calculation unit of the j + 1-th column, and send the target absolute value difference to the data stacking unit corresponding to the absolute value unit; in response to j = J, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the absolute value calculation unit of the j - 1-th column, and is configured to determine the target absolute value difference between the target depth value and the reference depth value, and is configured to send the target absolute value difference to the data stacking unit corresponding to the absolute value unit, and delete the target depth value.

[0109] According to an embodiment of the present disclosure, M in the M-th target depth value can be preset. For example, M = 1, that is, the first target depth value received by the absolute value calculation matrix is used as the reference depth value.

[0110] According to an embodiment of the present disclosure, after determining the reference depth value, the absolute value calculation unit configured to load the reference depth value can send the reference depth value to each absolute value calculation unit in the absolute value calculation matrix.

[0111] According to an embodiment of the present disclosure, the absolute value cache unit includes a first selector, at least two cache subunits, and a second selector; the first selector is configured to determine a first cache subunit from at least two cache subunits, where the first cache subunit is configured to cache the written stacking data; the second selector is configured to determine a second cache subunit from at least two absolute value cache units, and the second cache subunit is configured to provide the stacking data for the asynchronous buffer unit to read the stacking data.

[0112] According to an embodiment of the present disclosure, by making the absolute value cache unit include a first selector, at least two cache subunits, and a second selector, simultaneous reading and writing can be achieved, and the data reading and writing efficiency can be improved.

[0113] According to an embodiment of the present disclosure, the depth coefficient calculation module 104 further includes: an asynchronous buffer unit.

[0114] An asynchronous buffer unit is used to retrieve at least one target stack data from the absolute value cache unit corresponding to the asynchronous buffer unit, and temporarily store at least one target stack data according to the storage capacity threshold of the asynchronous buffer unit.

[0115] According to embodiments of this disclosure, the input feature map includes at least one input feature value; the pre-calculation module 105 includes a lookup unit and a pre-calculation unit; the lookup unit is used to obtain the target absolute value difference from at least one target stacked data stored in the asynchronous buffer unit based on a preset data volume, and is used to determine the depth coefficient corresponding to the target absolute value difference from a preset lookup table based on the target absolute value difference; the pre-calculation unit is used to multiply the depth coefficient with the corresponding input feature value to obtain the pre-calculation sub-result corresponding to the depth coefficient, and is used to obtain the pre-calculation result with the pre-calculation sub-result corresponding to at least one depth coefficient, and is used to send the pre-calculation result to the pulsating array 106.

[0116] According to embodiments of this disclosure, there is a one-to-one correspondence between the input feature values ​​in the input feature map and the depth values ​​in the depth map. That is, the input feature values ​​represent the pixel values ​​of each pixel, while the depth values ​​represent the depth values ​​of each pixel. In other words, the pixel values ​​and depth coefficients of the same pixel can be pre-calculated. Since there is a correspondence between depth values ​​and pixels, and also between depth coefficients and depth values, the correspondence between depth coefficients and pixels can be determined, as well as the input feature values ​​corresponding to those depth coefficients.

[0117] According to embodiments of this disclosure, the pre-computation unit may include a register If_reg for storing input feature values, a register reg for storing depth coefficients, and a multiplier. Figure 3 In the first pixel, fmap_0, Ifmap_1, and Ifmap_2 represent the input feature values ​​of the zeroth pixel, the first pixel, and the second pixel, respectively. In the second pixel, dx_0, dx_1, and dx_2 represent the depth coefficients of the zeroth pixel, the first pixel, and the second pixel, respectively. In the third pixel, pre_0, pre_1, and pre_2 represent the pre-computed sub-results of the zeroth pixel, the first pixel, and the second pixel, respectively.

[0118] According to embodiments of this disclosure, the pre-calculation between the depth coefficient and the input feature value can be multiplication. After multiplication, a pre-calculated sub-result for the pixel can be obtained, and there is a correspondence between the pre-calculated sub-result and the depth coefficient or the input feature value.

[0119] According to embodiments of this disclosure, the number of multipliers depends on the absolute value calculation matrix. For example, a 3x3 absolute value calculation matrix corresponds to a pre-computation using 3x1 multipliers.

[0120] According to an embodiment of this disclosure, based on depth similarity, the depth-aware convolution formula is as shown in the following formula (2).

[0121] (2)

[0122] According to the embodiments of this disclosure, it can be seen from formula (2) that pixels with a similar depth to the convolution center have a greater impact on the output during the convolution process. Therefore, in the depth-aware convolution mode, the pre-computation module 105 can replace the original input feature map in ordinary convolution after preprocessing the depth coefficients and the input feature map. Thus, inputting the pre-computation result into the systolic array 106 can realize the reuse of the systolic array 106. Whether it is the depth-aware convolution mode or the conventional convolution mode, a single systolic array 106 can be used to implement the convolution process.

[0123] According to an embodiment of this disclosure, after processing by the pre-computation module 105, the depth-aware convolution operation is converted into the output of a standard 2D convolution, as shown in formula (3).

[0124] (3)

[0125] Where R represents the local grid around p0 in x, w is the convolution kernel, and p0 and pn refer to the 0th and nth pixels, respectively.

[0126] According to embodiments of this disclosure, computational reuse can therefore be achieved using the systolic array 106. The input feature map pre-computation module 105 uses a multiplier to multiply the input feature map by the depth coefficients, which is equivalent to the input feature map input to the systolic array 106 after fine-tuning in a regular convolution mode. The pre-computation result is loaded into the PE cell register of the systolic array 106 for subsequent convolution calculations.

[0127] According to an embodiment of the present disclosure, the depth value caching module 103 includes a third selector and a depth value caching unit; the depth value caching unit is used to cache at least one depth value; when the working mode is a regular convolution mode, the third selector is used to output a depth value equal to a preset value; when the working mode is a depth-aware convolution mode, the selector is used to output at least one depth value from the caching unit in sequence.

[0128] According to embodiments of this disclosure, the depth value caching unit can perform depth value caching only after receiving a depth value loading enable signal sent by the depth value loading unit, and the depth value can be obtained from the external memory 111, which can be an external hard disk, disk or other device.

[0129] According to embodiments of this disclosure, under the control of a depth value determination enable signal and a depth value transmission enable signal, the depth value caching module 103 determines the depth value corresponding to the current working mode and transmits it.

[0130] According to embodiments of this disclosure, the depth value cache unit in the depth value cache module 103 may be composed of a set of BRAM (Block Random Access Memory), the size of which is determined by at least one depth value, i.e., the size of the depth map. The depth map is used to provide real-time depth coefficients for use by depth-aware convolution, and the size of the depth map to be cached is the same as that of the input feature map.

[0131] According to the embodiments of this disclosure, the preset value is not limited and can be any value that will not affect the conventional convolution mode, such as: the preset value can be 1, etc.

[0132] According to an embodiment of this disclosure, specifically, when in depth-aware convolution mode, the configurable mode module controls the depth value caching module 103 to normally output depth values ​​and send them to the depth coefficient calculation module 104; at the same time, the configurable mode module can also control the pre-calculation module 105 to normally output the pre-calculation results to the pulsating array 106 to complete the depth-aware convolution calculation.

[0133] According to embodiments of this disclosure, when the architecture operates in conventional convolution mode, the configurable mode module controls the depth value cache module 103 to output a preset value, such as 1, and sends it to the depth coefficient calculation module 104, ensuring that the effective range of the enable signal is the same as when calculating depth-aware convolution. In this case, the pre-calculation result output by the pre-calculation module 105 is the original value of the input feature map, and the systolic array 106 only loads the original feature map data and convolution kernels for ordinary convolution calculation.

[0134] According to embodiments of this disclosure, the configurable depth-aware convolutional operator hardware architecture can be made to run in both conventional convolutional mode and depth-aware convolutional mode by controlling the depth value output of the depth value caching module 103, and the various modules can be made compatible with different modes.

[0135] According to embodiments of this disclosure, the configurable depth-aware convolution operator hardware architecture further includes an input feature map caching module 108 and a weight caching module.

[0136] The input feature map caching module 108 is used to cache the input feature map and to send the input feature map to the pre-calculation module 105 when it receives the input data transmission enable signal sent by the task scheduling module 101.

[0137] The weight caching module is used to cache at least one weight value and to send at least one weight value to the systolic array 106 when a weight transmission enable signal is received from the task scheduling module 101, so that the systolic array 106 performs convolution processing on the pre-calculated result based on at least one weight value to obtain the convolution processing result.

[0138] According to embodiments of this disclosure, the input feature map caching module 108 may consist of a set of BRAMs, the number of BRAMs used depending on the size of the divided input feature maps. The weight caching module may also consist of a set of BRAMs, the number of BRAMs used depending on the size of the convolution window.

[0139] According to embodiments of this disclosure, the configurable depth-aware convolutional operator hardware architecture further includes a cache data loading module 107; the cache data loading module 107 includes an input feature map loading unit, a depth value loading unit, and a weight loading unit.

[0140] The input feature map loading unit is used to load at least one depth value into the depth value cache module 103 when receiving an input feature map loading enable signal sent by the task scheduling module 101; the depth value loading unit is used to load the input feature map into the input feature map cache module 108 when receiving a depth value loading enable signal sent by the task scheduling module 101; and the weight loading unit is used to load at least one weight value into the weight cache module when receiving a weight loading enable signal sent by the task scheduling module 101.

[0141] According to embodiments of this disclosure, the configurable depth-aware convolution operator hardware architecture may further include external memory 111 and external memory controller 112.

[0142] External memory 111 is used to store at least one depth value, an input feature map, and at least one weight value.

[0143] The external storage controller 112 is used to control the external memory 111 in order to retrieve at least one depth value, an input feature map, and at least one weight value from the external memory 111.

[0144] According to embodiments of this disclosure, the external memory 111 may be a Double Data Rate Synchronous Dynamic Random Access Memory (DDR).

[0145] According to embodiments of this disclosure, the cache data loading module 107 is primarily responsible for loading data from DDR and writing it into the on-chip BRAM module for use by the convolution calculation unit or other processing units. The BRAM module may include: a depth value cache module 103, an input feature map cache module 108, and a weight value cache module 109.

[0146] According to embodiments of this disclosure, the cache data loading module 107 integrates the functions of the input feature map loading unit, the depth value loading unit, and the weight loading unit, and uniformly manages the write enable signal of the BRAM to ensure the correct writing and caching of data.

[0147] According to embodiments of this disclosure, specifically, the input feature map loading module is mainly responsible for controlling the BRAM write enable, writing data from the DDR into the input feature map cache module BRAM. Specifically, taking a three-row input feature map as an example, two rows of BRAM in the input feature map cache module 108 are needed. When the first row of input feature values ​​arrives, the write enable signal of the first row of BRAM is pulled high (set to 1), and one pixel value is written each clock cycle. When the first row of BRAM is full, the write enable signal of the second row of BRAM is pulled high, and the pixels of the first row are written into the second row of BRAM. At this time, the write enable signal of the first row of BRAM remains high, continuously writing pixels of the second row of input feature maps. When the second row of BRAM is full, the next clock cycle simultaneously reads the input and two BRAM data to obtain the three rows of input feature maps.

[0148] According to embodiments of this disclosure, the depth map loading module is responsible for writing data from the DDR into the BRAM of the depth value cache module 103 and controlling the write enable signal of the BRAM. The weight loading module is mainly responsible for loading the weight values ​​of the convolution kernel from the DDR and writing them into the BRAM of the weight cache module for use by the convolution calculation unit. That is, the cache data loading module 107 ensures that the weight data is written correctly by controlling the write enable signal of the BRAM.

[0149] According to embodiments of this disclosure, the configurable depth-aware convolution operator hardware architecture further includes: a post-module 110; the post-module 110 is used to perform pooling processing on the convolution processing result to obtain pooled data, and to perform mapping processing on the pooled data to obtain target output data.

[0150] According to embodiments of this disclosure, the back module 110 may include any units that can be added after convolution, such as pooling units, mapping units, convolution and normalization units, and activation function processing units.

[0151] According to embodiments of this disclosure, the target output data can be used in the next step or stored in a memory such as DDR.

[0152] According to embodiments of this disclosure, a pooling unit can perform pooling processing on the convolution results. A mapping unit, which can be a fully connected neural network placed after the pooling unit, maps the pooled convolution results to the desired target output data.

[0153] Figure 4 A schematic diagram illustrating the task processing flowchart of a configurable depth-aware convolution operator hardware architecture according to an embodiment of the present disclosure is provided.

[0154] like Figure 4 As shown, the task processing flowchart includes operations S401 to S407.

[0155] When operating S401, the task scheduling module receives working mode information.

[0156] In operation S402, the task scheduling module determines whether the working mode is depth-aware convolution mode. If the working mode is determined to be regular convolution mode, operation S403 is executed; if the working mode is determined to be depth-aware convolution mode, operation S405 is executed.

[0157] During operation S403, the task scheduling module sends a mode enable signal to the configurable module, sends an input feature map loading enable signal to the input feature map loading unit, and sends a weight loading enable signal to the weight loading unit.

[0158] According to embodiments of this disclosure, a mode enable signal is sent to a configurable module so that the configurable module can send a control depth value caching module to determine the depth value corresponding to the working mode, an input feature map loading enable signal is sent to an input feature map loading unit so that the input feature map loading unit can control the input feature map caching module to cache the input feature value, and a weight loading enable signal is sent to a weight loading unit so that the weight loading unit can control the weight caching unit to cache the weight value.

[0159] In operation S404, when the task scheduling module determines that the input feature value cached by the feature map cache module meets the first cache condition, it sends an input data sending enable signal so that the input feature map cache module sends the input feature map to the pre-computation module; when the task scheduling module determines that the weight value cached by the weight loading unit meets the second cache condition, it sends a weight value sending enable signal to the pulsating array 106; when the task scheduling module determines that the depth value cache module has determined the depth value, it sends a depth value sending enable signal to the depth value cache module so that the depth value cache module sends the depth value to the depth coefficient calculation module.

[0160] During operation S405, the task scheduling module sends a depth value loading enable signal to the depth value loading unit, sends a mode enable signal to the configurable module, sends an input feature map loading enable signal to the input feature map loading unit, and sends a weight loading enable signal to the weight loading unit.

[0161] According to embodiments of this disclosure, the task scheduling module sends a depth value loading enable signal to the depth value loading unit so that the depth value loading unit controls the depth value caching module to cache depth values; sends a mode enable signal to the configurable module so that the configurable module sends a control to the depth value caching module to determine the depth value corresponding to the working mode; sends an input feature map loading enable signal to the input feature map loading unit so that the input feature map loading unit controls the input feature map caching module to cache input feature values; and sends a weight loading enable signal to the weight loading unit so that the weight loading unit controls the weight caching unit to cache weight values.

[0162] In operation S406, when the task scheduling module determines that the input feature value cached by the input feature map cache module meets the first cache condition, it sends an input data sending enable signal so that the input feature map cache module sends the input feature map to the pre-computation module; when the task scheduling module determines that the weight value cached by the weight value cache unit meets the second cache condition, it sends a weight value sending enable signal to the pulsating array 106; when the task scheduling module determines that the weight value cached by the depth value cache module meets the third cache condition and the depth value cache module has determined the depth value, it sends a depth value sending enable signal to the depth value cache module so that the depth value cache module sends the depth value to the depth coefficient calculation module.

[0163] According to embodiments of this disclosure, the first cache condition is determined based on the BRAM in the input feature map cache module 108. For example, if it is determined that each row of the BRAM stores an input feature value, the first cache condition is considered to be satisfied. The second cache condition is determined based on the BRAM in the weight value cache unit. The third cache condition is determined based on the BRAM in the depth value cache module 103.

[0164] During operation of S407, the depth coefficient calculation module, the pre-calculation module, and the pulsation array 106 are started sequentially for processing.

[0165] According to the embodiments of this disclosure, the task scheduling module 101 can send a corresponding module start enable signal to start the module based on the data received by the depth coefficient calculation module 104, the pre-calculation module 105 and the pulsation array 106, when it is determined that the module has received the data required for calculation. For example, when the depth coefficient calculation module 104 receives the depth value, it can start the depth coefficient calculation module 104 to execute the execution logic of the module.

[0166] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0167] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0168] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A configurable depth-aware convolution operator hardware architecture, characterized in that, Comprising: A task scheduling module, configured to generate a data enable signal and control the execution order of convolution tasks based on the data enable signal, wherein the data enable signal includes a mode enable signal and a depth value transmission enable signal; A configurable module, configured to determine a working mode based on the mode enable signal and send a depth value determination enable signal corresponding to the working mode to a depth value cache module, wherein the working mode includes a regular convolution mode or a depth-aware convolution mode; The depth value cache module is configured to determine at least one depth value corresponding to the working mode when receiving the depth value determination enable signal, and send at least one depth value corresponding to the working mode to a depth coefficient calculation module respectively when receiving the depth value transmission enable signal; The depth coefficient calculation module is configured to obtain depth coefficients corresponding to at least one of the depth values based on at least one of the depth values and send the depth coefficients corresponding to at least one of the depth values to a pre-calculation module; The pre-calculation module is configured to perform pre-calculation on the depth coefficients corresponding to at least one of the depth values and an input feature map to obtain a pre-calculation result and send the pre-calculation result to a systolic array; The systolic array is configured to perform convolution processing on the pre-calculation result to obtain a convolution processing result.

2. The configurable depth-aware convolution operator hardware architecture according to claim 1, characterized in that, The depth coefficient calculation module includes: an absolute value calculation matrix, at least one data stacking unit, and at least one absolute value cache unit; The absolute value calculation matrix is configured to determine a reference depth value from at least one of the depth values and determine the absolute value difference between the reference depth value and each of at least one of the depth values; At least one of the data stacking units, each of the data stacking units corresponding to at least one target absolute value calculation unit in the absolute value calculation matrix, the data stacking unit being configured to splice the absolute value differences output by at least one of the target absolute value calculation units in the same system clock cycle to obtain stacked data, wherein at least one of the target absolute value calculation units is located in the same row; At least one of the absolute value cache units, respectively connected to at least one of the data stacking units, the absolute value cache unit being configured to cache the stacked data.

3. The configurable depth-aware convolution operator hardware architecture according to claim 2, characterized in that, The absolute value calculation matrix includes J columns of absolute value calculation units; When in the i-th system clock cycle and i is not equal to N*J, In response to j = 1, the absolute value calculation unit in the j-th column of the absolute value calculation matrix is configured to load a target depth value from at least one of the depth values sent by the depth value cache module and send the target depth value to the absolute value calculation unit in the j+1-th column, wherein i is an integer greater than or equal to 1, and N, J are integers; In response to 1 < j < J, the absolute value calculation unit in the j-th column is configured to load the target depth value sent by the absolute value calculation unit in the j-1-th column and send the target depth value to the absolute value calculation unit in the j+1-th column; In response to j = J, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the absolute value calculation unit of the (j - 1)-th column; In the case where it is the i-th system clock cycle and i is equal to N * J, In response to j = 1, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the depth value cache module, and is configured to determine the target absolute value difference between the target depth value and the reference depth value, and is configured to send the target depth value to the absolute value calculation unit of the (j + 1)-th column, and send the target absolute value difference to the data stacking unit corresponding to the absolute value calculation unit, where the reference depth value is the M-th target depth value sent by the depth value cache module, and M is an integer greater than or equal to 1; In response to 1 < j < J, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the absolute value calculation unit of the (j - 1)-th column, and is configured to determine the target absolute value difference between the target depth value and the reference depth value, and is configured to send the target depth value to the absolute value calculation unit of the (j + 1)-th column, and send the target absolute value difference to the data stacking unit corresponding to the absolute value calculation unit; In response to j = J, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the absolute value calculation unit of the (j - 1)-th column, and is configured to determine the target absolute value difference between the target depth value and the reference depth value, and is configured to send the target absolute value difference to the data stacking unit corresponding to the absolute value calculation unit, and delete the target depth value.

4. The configurable depth-aware convolution operator hardware architecture according to claim 2, characterized in that, The absolute value cache unit includes a first selector, at least two cache sub-units, and a second selector; The first selector is configured to determine a first cache sub-unit from at least two cache sub-units, where the first cache sub-unit is configured to cache the written stacking data; The second selector is configured to determine a second cache sub-unit from at least two absolute value cache units, and the second cache sub-unit is configured to provide the stacking data so that the asynchronous buffer unit reads the stacking data.

5. The configurable depth-aware convolution operator hardware architecture according to claim 4, characterized in that, The depth coefficient calculation module further includes: the asynchronous buffer unit; The asynchronous buffer unit is configured to obtain at least one target stacking data from the absolute value cache unit corresponding to the asynchronous buffer unit, and temporarily store at least one of the target stacking data according to the storage capacity threshold of the asynchronous buffer unit.

6. The configurable depth-aware convolution operator hardware architecture according to claim 5, characterized in that, The input feature map includes at least one input feature value; the pre-calculation module includes a lookup unit and a pre-calculation unit; The lookup unit is configured to obtain a target absolute value difference from at least one of the target stacking data stored in the asynchronous buffer unit based on a preset data volume, and is configured to determine a depth coefficient corresponding to the target absolute value difference from a preset lookup table based on the target absolute value difference; The pre-computation unit is configured to multiply the depth coefficient with the corresponding input feature value to obtain a pre-computation sub-result corresponding to the depth coefficient, and to obtain a pre-computation result by combining the pre-computation sub-result corresponding to at least one depth coefficient, and to send the pre-computation result to the pulsation array.

7. The configurable depth-aware convolution operator hardware architecture according to claim 1, characterized in that, The depth value caching module includes a third selector and a depth value caching unit; The depth value caching unit is used to cache at least one of the depth values; When the working mode is the regular convolution mode, the third selector is used to output a depth value equal to a preset value; When the operating mode is the depth-aware convolution mode, the third selector is used to output at least one of the depth values ​​from the depth value cache unit in sequence.

8. The configurable depth-aware convolution operator hardware architecture according to claim 1, characterized in that, The configurable depth-aware convolution operator hardware architecture also includes: an input feature map caching module and a weight caching module; The input feature map caching module is used to cache the input feature map and to send the input feature map to the pre-computation module when an input data transmission enable signal is received from the task scheduling module. The weight caching module is used to cache at least one weight value and, upon receiving a weight transmission enable signal sent by the task scheduling module, to send at least one weight value to the systolic array, so that the systolic array performs convolution processing on the pre-calculated result based on at least one weight value to obtain a convolution processing result.

9. The configurable depth-aware convolution operator hardware architecture according to claim 8, characterized in that, The configurable depth-aware convolutional operator hardware architecture also includes a cached data loading module; the cached data loading module includes an input feature map loading unit, a depth value loading unit, and a weight loading unit. The input feature map loading unit is used to load at least one depth value into the depth value cache module when it receives an input feature map loading enable signal sent by the task scheduling module. The depth value loading unit is used to load the input feature map into the input feature map cache module when it receives the depth value loading enable signal sent by the task scheduling module; The weight loading unit is used to load at least one weight value into the weight cache module when it receives a weight loading enable signal sent by the task scheduling module.

10. The configurable depth-aware convolution operator hardware architecture according to claim 1, characterized in that, Also includes: Post-processing module; The post-processing module is used to perform pooling processing on the convolution processing result to obtain pooled data, and to perform mapping processing on the pooled data to obtain target output data.

Images