Configurable depth-aware convolution operator hardware architecture

Abstract

The invention provides a configurable depth perception convolution operator hardware architecture which can be applied to the technical field of hardware design. Comprising a task scheduling module used for generating a data enable signal and controlling the execution sequence of convolution tasks based on the data enable signal; the configurable module is used for determining a working mode based on the mode enable signal; the depth value cache module is used for determining at least one depth value corresponding to the working mode under the condition that the depth value determination enable signal is received, and respectively sending the at least one depth value corresponding to the working mode to the depth coefficient calculation module under the condition that the depth value sending enable signal is received; the depth coefficient calculation module is used for obtaining a depth coefficient set corresponding to the at least one depth value based on the at least one depth value; the pre-calculation module is used for pre-calculating the depth coefficient set and the input feature map to obtain a pre-calculation result; and the systolic array is used for performing convolution processing on the pre-calculation result to obtain a convolution processing result.

Description

Technical Field

[0001] The present disclosure relates to the technical field of hardware design, and in particular to a configurable depth-aware convolution operator hardware architecture. Background Art

[0002] In recent years, Convolutional Neural Networks (CNN) have been widely used in the field of computer vision, especially in image classification, object detection and other tasks. CNN extracts the spatial features of images through two-dimensional convolution operations and is suitable for most scenarios. However, with the development of depth sensors, the combination of depth images and RGB images can provide more geometric information and improve the accuracy of scene understanding. However, due to the fixed grid structure, traditional convolutional neural networks are difficult to effectively handle the geometric variability in images, which limits their use of depth information.

[0003] To solve this problem, researchers proposed a depth-aware convolution operator, which introduces depth similarity terms into traditional convolutional neural networks so that pixels with similar depths have a greater impact on the convolution results. Compared with two-stream networks or 3D convolutions, depth-aware convolutions can flexibly integrate geometric information and improve the ability to understand the geometric structure of the scene without significantly increasing computational complexity. However, during the research process, the inventors found that the hardware architecture in the relevant technologies is mostly designed with traditional convolutional neural networks, which cannot give full play to the advantages of the depth-aware convolution operator and reduces the application efficiency of the depth-aware convolution operator. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a configurable depth-aware convolution operator hardware architecture.

[0005] According to one aspect of the present disclosure, a configurable depth-aware convolution operator hardware architecture is provided, including:

[0006] A task scheduling module, used to generate a data enable signal and control the execution order of the convolution task based on the data enable signal, wherein the data enable signal includes a mode enable signal and a depth value sending enable signal;

[0007] A configurable module, configured to determine an operating mode based on a mode enable signal, and send a depth value determination enable signal corresponding to the operating mode to a depth value buffer module, wherein the operating mode includes a conventional convolution mode or a depth-aware convolution mode;

[0008] A depth value buffer module, configured to determine at least one depth value corresponding to the working mode when a depth value determination enable signal is received, and to send at least one depth value corresponding to the working mode to the depth coefficient calculation module when a depth value sending enable signal is received;

[0009] A depth coefficient calculation module, configured to obtain depth coefficients corresponding to at least one depth value based on the at least one depth value, and send the depth coefficients corresponding to the at least one depth value to a pre-calculation module;

[0010] A pre-calculation module, configured to perform pre-calculation on the depth coefficients corresponding to the at least one depth value and an input feature map to obtain a pre-calculation result, and send the pre-calculation result to a systolic array;

[0011] A systolic array, configured to perform convolution processing on the pre-calculation result to obtain a convolution processing result.

[0012] According to an embodiment of the present disclosure, the depth coefficient calculation module includes: an absolute value calculation matrix, at least one data stacking unit, and at least one absolute value cache unit;

[0013] The absolute value calculation matrix is configured to determine a reference depth value from the at least one depth value, and determine the absolute value difference between the reference depth value and each of the at least one depth value;

[0014] Each of the at least one data stacking unit corresponds to at least one target absolute value calculation unit in the absolute value calculation matrix. The data stacking unit is configured to splice the absolute value differences output by the at least one target absolute value calculation unit in the same system clock cycle to obtain stacked data, where the at least one target absolute value calculation unit is located in the same row;

[0015] Each of the at least one absolute value cache unit is respectively connected to the at least one data stacking unit. The absolute value cache unit is configured to cache the stacked data.

[0016] According to an embodiment of the present disclosure, the absolute value calculation matrix includes J columns of absolute value calculation units;

[0017] In the case where it is the i-th system clock cycle and i is not equal to N*J,

[0018] In response to j = 1, the absolute value calculation unit in the j-th column of the absolute value calculation matrix is configured to load the target depth value among the at least one depth value sent by the depth value cache module, and send the target depth value to the absolute value calculation unit in the j+1-th column, where i is an integer greater than or equal to 1, and N and J are integers;

[0019] In response to 1 < j < J, the absolute value calculation unit in the j-th column is configured to load the target depth value sent by the absolute value calculation unit in the j-1-th column, and send the target depth value to the absolute value calculation unit in the j+1-th column;

[0020] In response to j = J, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the absolute value calculation unit of the (j - 1)-th column;

[0021] In the case where it is at the i-th system clock cycle and i is equal to N * J,

[0022] In response to j = 1, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the depth value cache module, and to determine the target absolute value difference between the target depth value and the reference depth value, and to send the target depth value to the absolute value calculation unit of the (j + 1)-th column, and send the target absolute value difference to the data stacking unit corresponding to the absolute value unit, wherein the reference depth value is the M-th target depth value sent by the depth value cache module, and M is an integer greater than or equal to 1;

[0023] In response to 1 < j < J, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the absolute value calculation unit of the (j - 1)-th column, and to determine the target absolute value difference between the target depth value and the reference depth value, and to send the target depth value to the absolute value calculation unit of the (j + 1)-th column, and send the target absolute value difference to the data stacking unit corresponding to the absolute value unit;

[0024] In response to j = J, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the absolute value calculation unit of the (j - 1)-th column, and to determine the target absolute value difference between the target depth value and the reference depth value, and to send the target absolute value difference to the data stacking unit corresponding to the absolute value unit, and delete the target depth value.

[0025] According to an embodiment of the present disclosure, the absolute value cache unit includes a first selector, at least two cache subunits, and a second selector;

[0026] The first selector is configured to determine a first cache subunit from at least two cache subunits, wherein the first cache subunit is configured to cache the written stacked data;

[0027] The second selector is configured to determine a second cache subunit from at least two absolute value cache units, and the second cache subunit is configured to provide the stacked data for the asynchronous buffer unit to read the stacked data.

[0028] According to an embodiment of the present disclosure, the depth coefficient calculation module further includes: an asynchronous buffer unit.

[0029] The asynchronous buffer unit is configured to obtain at least one target stacked data from the absolute value cache unit corresponding to the asynchronous buffer unit, and temporarily store the at least one target stacked data according to the storage capacity threshold of the asynchronous buffer unit.

[0030] According to an embodiment of the present disclosure, the input feature map includes at least one input feature value; the pre-calculation module includes a search unit and a pre-calculation unit;

[0031] a search unit, configured to obtain a target absolute value difference from at least one target stacked data stored in the asynchronous buffer unit based on a preset data amount, and to determine a depth coefficient corresponding to the target absolute value difference from a preset search table based on the target absolute value difference;

[0032] A pre-calculation unit is used to multiply the depth coefficient with the corresponding input eigenvalue to obtain a pre-calculation sub-result corresponding to the depth coefficient, and to obtain a pre-calculation result from the pre-calculation sub-result corresponding to at least one depth coefficient, and to send the pre-calculation result to the systolic array.

[0033] According to an embodiment of the present disclosure, the depth value cache module includes a third selector and a depth value cache unit;

[0034] The depth value cache unit is used to cache at least one depth value;

[0035] When the working mode is the normal convolution mode, the third selector is used to output a depth value equal to a preset value;

[0036] When the working mode is the depth-aware convolution mode, the third selector is used to output at least one depth value in the depth value cache unit in sequence.

[0037] According to an embodiment of the present disclosure, the configurable depth-aware convolution operator hardware architecture further includes: an input feature map cache module and a weight cache module;

[0038] An input feature map cache module, used for caching the input feature map, and for sending the input feature map to the pre-calculation module when receiving an input data sending enable signal sent by the task scheduling module;

[0039] The weight cache module is used to cache at least one weight value, and is used to send at least one weight value to the systolic array when receiving a weight sending enable signal sent by the task scheduling module, so that the systolic array performs convolution processing on the pre-calculated result based on the at least one weight value to obtain a convolution processing result.

[0040] According to an embodiment of the present disclosure, the configurable depth-aware convolution operator hardware architecture further includes a cache data loading module; the cache data loading module includes an input feature map loading unit, a depth value loading unit, and a weight loading unit;

[0041] An input feature map loading unit, configured to load at least one depth value into a depth value cache module upon receiving an input feature map loading enable signal sent by a task scheduling module;

[0042] A depth value loading unit, configured to load the input feature map into the input feature map cache module upon receiving a depth value loading enable signal sent by the task scheduling module;

[0043] The weight loading unit is used to load at least one weight value into the weight cache module when receiving a weight loading enable signal sent by the task scheduling module.

[0044] According to an embodiment of the present disclosure, the configurable depth-aware convolution operator hardware architecture further includes: a post-processing module;

[0045] The post-processing module is used to perform pooling processing on the convolution processing results to obtain pooled data, and to perform mapping processing on the pooled data to obtain target output data.

[0046] According to the configurable depth-aware convolution operator hardware architecture disclosed in the present invention, a data enable signal is generated by a task scheduling module to control the execution order of the convolution task, such as generating a mode enable signal. The configurable module determines the working mode through the mode enable signal, and controls the depth value cache module to output at least one depth value that satisfies the working mode, so that the depth coefficient calculated by the at least one depth value satisfies the calculation requirements under the working mode, and the depth coefficient and the input feature map are pre-calculated by the pre-calculation module, and the obtained pre-calculation result is sent to the systolic array for convolution processing, so that the systolic array can be used normally in different working modes, thereby at least partially solving the technical problem that the hardware architecture in the related art cannot give full play to the advantages of the depth-aware convolution operator, and realizing the flexible switching of the hardware system between the conventional convolution mode and the depth-aware convolution mode, so as to better give play to the advantages of the depth-aware convolution operator, and in the implementation process, the conventional convolution mode and the depth-aware convolution data flow are basically the same, and the data flow and each module can be shared while being compatible with different working modes, thereby reducing the complexity of data flow scheduling and saving certain hardware resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0048] Figure 1 The schematic diagram of the hardware architecture of a configurable depth-aware convolution operator according to the first embodiment of the present disclosure is shown schematically;

[0049] Figure 2 The schematic diagram of the hardware architecture of a configurable depth-aware convolution operator according to the second embodiment of the present disclosure is shown;

[0050] Figure 3Schematically shows a schematic diagram of the architecture of a depth coefficient calculation module and a pre-calculation module according to a second embodiment of the present disclosure;

[0051] Figure 4 The figure schematically shows a task processing flow chart of a configurable depth-aware convolution operator hardware architecture according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0053] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0054] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0055] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0056] The embodiment of the present disclosure provides a configurable depth-aware convolution operator hardware architecture, characterized in that it includes: a task scheduling module, which is used to generate a data enable signal and control the execution order of the convolution task based on the data enable signal, wherein the data enable signal includes a mode enable signal and a depth value sending enable signal; a configurable module, which is used to determine the working mode based on the mode enable signal, and send a depth value determination enable signal corresponding to the working mode to a depth value buffer module, wherein the working mode includes a conventional convolution mode or a depth-aware convolution mode; a depth value buffer module, which determines at least one depth value corresponding to the working mode when receiving the depth value determination enable signal, and sends at least one depth value corresponding to the working mode to a depth coefficient calculation module when receiving the depth value sending enable signal; a depth coefficient calculation module, which is used to obtain a depth coefficient set corresponding to at least one depth value based on at least one depth value, and send the depth coefficient set to a pre-calculation module; a pre-calculation module, which is used to pre-calculate the depth coefficient set and an input feature map to obtain a pre-calculation result, and send the pre-calculation result to a systolic array; and a systolic array, which is used to perform convolution processing on the pre-calculation result to obtain a convolution processing result.

[0057] Figure 1 The figure schematically shows a hardware architecture diagram of a configurable depth-aware convolution operator according to the first embodiment of the present disclosure.

[0058] like Figure 1 As shown, the configurable depth-aware convolution operator hardware architecture includes the following modules.

[0059] The task scheduling module 101 is used to generate a data enable signal and control the execution order of the convolution task based on the data enable signal, wherein the data enable signal includes a mode enable signal and a depth value sending enable signal.

[0060] The configurable module 102 is used to determine the working mode based on the mode enable signal and send a depth value determination enable signal corresponding to the working mode to the depth value cache module 103, wherein the working mode includes a conventional convolution mode or a depth-aware convolution mode.

[0061] The depth value cache module 103 is used to determine at least one depth value corresponding to the working mode when a depth value determination enable signal is received, and to send at least one depth value corresponding to the working mode to the depth coefficient calculation module 104 when a depth value sending enable signal is received.

[0062] The depth coefficient calculation module 104 is configured to obtain a depth coefficient corresponding to the at least one depth value based on the at least one depth value, and send the depth coefficient corresponding to the at least one depth value to the pre-calculation module 105 .

[0063] The pre-calculation module 105 is used to pre-calculate the depth coefficient corresponding to at least one depth value and the input feature map to obtain a pre-calculation result, and send the pre-calculation result to the systolic array 106.

[0064] The systolic array 106 is used to perform convolution processing on the pre-calculated result to obtain a convolution processing result.

[0065] According to an embodiment of the present disclosure, Figure 1 The dotted line in the figure can represent the signal transmission direction, and the solid line can represent the data transmission direction.

[0066] According to an embodiment of the present disclosure, the task scheduling module 101 may determine the current working mode according to the received working mode signal, and send a mode enable signal corresponding to the working mode to the configurable module 102 .

[0067] According to an embodiment of the present disclosure, the task scheduling module 101 is used to coordinate the task execution order of each module to ensure that data processing and calculation can be performed sequentially. Specifically, the task scheduling module 101 is used to generate a scheduling plan based on task priority and the availability of computing resources to ensure that convolution computing tasks are performed in sequence. There is no limitation on the generation order and generation timing of each data enable signal, and the generation order and generation timing of different data enable signals can be determined according to requirements.

[0068] According to an embodiment of the present disclosure, the task scheduling module 101 is used to generate a data enable signal to ensure that data is loaded from a specified module to another module at a specific time. Exemplarily, the task scheduling module 101 generates an enable signal for loading input feature maps, weights, and depth map data. In the depth perception convolution mode: first schedule the input feature map and depth value to pre-calculate the depth perception coefficient, and then schedule the weight value to the systolic array 106. In the normal perception convolution mode, since the depth value cache module 103 outputs a preset value for subsequent calculations, the depth value transmission does not need to occupy the bus bandwidth. It is only necessary to schedule the input feature map data and weight data and deploy them to the systolic array 106 for normal convolution operations. When the data enters the depth coefficient calculation module 104, the calculation enable signal needs to be controlled to be enabled after the data completely fills the calculation array.

[0069] According to an embodiment of the present disclosure, the configurable module 102 can be used to determine the current operating mode based on the mode enable signal, and to control the register included in the configurable module 102 to output 1 or 0 to characterize the current operating mode. For example, mode=1 can represent working in depth perception convolution mode, and mode=0 represents working in conventional convolution mode.

[0070] According to an embodiment of the present disclosure, the configurable mode module can control the direction of data flow by configuring registers according to system requirements to achieve switching between normal convolution and depth-aware convolution modes.

[0071] According to an embodiment of the present disclosure, at least one depth value is used to construct a depth map, and the depth map includes at least one depth value.

[0072] According to an embodiment of the present disclosure, the depth value cache module 103 can be used to determine the output command indicated by the enable signal based on the depth value, and output at least one depth value corresponding to the working mode, for example: when the working mode is the depth perception convolution mode, the depth value of the depth map present in the depth value cache module 103 is output, and when the working mode is the depth perception convolution mode, at least one preset value is output.

[0073] According to an embodiment of the present disclosure, the depth coefficient calculation module 104 is used to dynamically calculate the depth coefficient required in the convolution process based on at least one depth value.

[0074] According to an embodiment of the present disclosure, the input feature map pre-calculation module 105 is used to pre-process the input feature values ​​and depth coefficients included in the input feature map.

[0075] According to an embodiment of the present disclosure, the systolic array 106 works in coordination through multiple parallel processing units PE to implement parallel convolution calculations of input feature maps, depth maps, and weight data. Each processing unit is responsible for a specific computing task and ultimately outputs a convolution result. PE can be a computing device such as a processor, a computer, or a server.

[0076] According to an embodiment of the present disclosure, the depth coefficient calculation module 104 performs dynamic depth coefficient calculation according to the depth map, and transmits the calculated depth coefficient to the input pre-calculation module 105. The pre-calculation module 105 performs preliminary processing on the input feature map and the depth coefficient, and provides a pre-processing result for the convolution operation of the systolic array 106.

[0077] According to an embodiment of the present disclosure, the task scheduling module 101 generates a data enable signal to control the execution order of the convolution task, such as generating a mode enable signal, and the configurable module 102 determines the working mode through the mode enable signal, and controls the depth value cache module 103 to output at least one depth value that satisfies the working mode, so that the depth coefficient calculated by the at least one depth value satisfies the calculation requirements under the working mode, and the depth coefficient and the input feature map are pre-calculated by the pre-calculation module 105, and the obtained pre-calculation result is sent to the systolic array 106 for convolution processing, so that the systolic array 106 can be used normally in different working modes, thereby at least partially solving the technical problem that the hardware architecture in the related art cannot give full play to the advantages of the depth perception convolution operator, and realizing the flexible switching of the hardware system between the conventional convolution mode and the depth perception convolution mode, so as to better give play to the advantages of the depth perception convolution operator, and in the implementation process, the conventional convolution mode and the depth perception convolution data flow are basically the same, and the data flow and each module can be shared while being compatible with different working modes, thereby reducing the complexity of data flow scheduling and saving certain hardware resources.

[0078] Figure 2 A schematic diagram of a configurable depth-aware convolution operator hardware architecture according to a second embodiment of the present disclosure is shown schematically.

[0079] like Figure 2 As shown, the configurable depth-aware convolution operator hardware architecture includes the following modules: external memory 111, external storage controller 112, cache data loading module 107, input feature map cache module 108, depth value cache module 103, weight value cache module 109, task scheduling module 101, configurable module 102, depth coefficient calculation module 104, pre-calculation module 105, systolic array 106, and post-module 110.

[0080] According to an embodiment of the present disclosure, Figure 2 The dotted lines in the figure can represent the flow direction of control signals, and the solid lines can represent the flow direction of data.

[0081] According to an embodiment of the present disclosure, the configurable module 102 can make the configurable depth-aware convolution operator hardware architecture run a conventional convolution mode or a depth-aware convolution mode by configuring registers of each module. The mode switching is achieved by switching the data stream through a data selector.

[0082] According to the embodiments of the present disclosure, the hardware system can flexibly switch between the conventional convolution mode and the depth perception convolution mode. The architecture can reasonably configure computing resources and optimize computing performance in different application scenarios to meet diverse computing needs, and through the parallel computing characteristics and configurable mode modules of the systolic array 106, the systolic array 106 computing units and on-chip cache are fully utilized to achieve efficient data processing and convolution operations.

[0083] Figure 3 The schematic diagram of the architecture of the depth coefficient calculation module 104 and the pre-calculation module 105 according to the second embodiment of the present disclosure is shown schematically.

[0084] like Figure 3 As shown, the depth value cache module 103 may include a third selector, the depth coefficient calculation module 104 includes: an absolute value calculation matrix, at least one data stacking unit, at least one absolute value cache unit, and an asynchronous buffer unit; the pre-calculation module 105 includes a search unit and a pre-calculation unit.

[0085] According to an embodiment of the present disclosure, the absolute value calculation matrix is ​​used to calculate the depth difference of the input data, and the data stacking unit splices the results output by the absolute value calculation matrix. The spliced ​​data is stored in an absolute value cache unit including a RAM (Random Access Memory), and then quickly queried and processed by a search unit including a ROM (Read Only Memory). The entire module is coordinated by a clock signal to ensure accurate processing and efficient transmission of data.

[0086] According to an embodiment of the present disclosure, taking a 3*3 depth perception convolution operator as an example, a 3*3 absolute value calculation matrix is ​​required, and thus three third selectors, three data stacking units, three absolute value cache units, three asynchronous buffer units, three search units, and three pre-calculation units are adaptively required.

[0087] According to an embodiment of the present disclosure, at least one third selector can select an output depth value dp_1, dp_2, dp_3 or a preset value 1'b1 through a working mode mode.

[0088] According to an embodiment of the present disclosure, the depth map includes a depth value of each pixel, that is, one depth value corresponds to one pixel. In the depth-aware convolution, the depth similarity between each pixel is defined by the following formula (1).

[0089] (1)

[0090] in, is a constant, and Represents pixel points and The depth value of .

[0091] According to the embodiment of the present disclosure, in the first system clock cycle of starting the depth coefficient calculation module 104, the depth values ​​dp_1, dp_2, and dp_3 read from the depth value cache module 103 are sent as input to the register reg of the first-level calculation array, that is, the register reg of the absolute value calculation unit located in the first column. In each cycle, the data flows to the right in the horizontal direction of the absolute value calculation matrix and is loaded once. In the third system clock cycle, the calculation array is filled, wherein, according to Figure 3 reg loading, the depth value of the first row and third column can be the reference depth value D(p i ) for the remaining computing units to perform absolute value difference calculations.

[0092] According to an embodiment of the present disclosure, cal is a calculator in the absolute value calculation unit, and the calculator can be used to determine whether any depth value is equal to D(p i ), and is used to obtain the absolute value difference between the depth values ​​by subtracting the depth value with a larger absolute value from the depth value with a smaller absolute value.

[0093] According to the embodiments of the present disclosure, since the absolute value of the depth difference is calculated using combinatorial logic, the absolute value difference corresponding to the current pixel point will be stored in the output register out_reg in the same system clock cycle. In order for the depth coefficient module to complete the absolute value difference calculation of one pixel in one system clock cycle, the results of all current output registers must be stored in RAM within one cycle, so a data stacking unit is introduced to splice the absolute value differences of the same row into stacked data with 3 times the bit width and write it into the absolute value cache unit.

[0094] According to an embodiment of the present disclosure, the data stacking unit mainly includes at least one RAM and a first selector and a second selector, which adopts a ping-pong buffer structure. One row of RAM caches one row of stacked data. The RAM can be expressed as RAMk_n, where k represents the kth data stacking unit and n represents the number of RAMs in the first data stacking unit.

[0095] According to the embodiments of the present disclosure, since a deep coefficient table lookup is required later, one column of valid data is required to be transmitted per system clock cycle, and calculations need to be performed at three times the original clock frequency. Therefore, an asynchronous buffer unit is introduced. The asynchronous buffer unit can be an asynchronous FIFO (First Input First Output) buffer. There is no limit on the storage capacity threshold, which can be selected according to actual conditions, such as a depth of 16.

[0096] According to an embodiment of the present disclosure, by placing an asynchronous buffer unit after the absolute value cache unit, data loss due to too fast data writing can be avoided.

[0097] According to an embodiment of the present disclosure, the search unit may include a storage space, which may be a ROM. The ROM may store a preset lookup table, which includes the correspondence between the absolute value difference and the depth coefficient. The absolute value difference of the depth coefficient of the corresponding bit may be intercepted each time and sent to the ROM for table lookup. The quantized value of the e index stored in the ROM is output to the next level pre-calculation unit.

[0098] According to the embodiments of the present disclosure, placing ROM after RAM brings a benefit based on meeting the computing requirements: reducing the use of ROM resources. Specifically, if the method of looking up the table before RAM is adopted, 9 ROMs are required for the absolute value calculation matrix of 3*3, and the method of looking up the table after RAM only uses one third of the ROM, that is, the on-chip resources are optimized.

[0099] According to an embodiment of the present disclosure, the specific data transmission process between the depth coefficient calculation module 104 and the pre-calculation module 105 can be as follows. Taking the first row of absolute value calculation units in the absolute value calculation matrix as an example, the third selector located in the first row determines the depth value selected by the enable signal according to the depth value to be output. The depth value includes at least one depth value capable of representing the depth map stored in the depth value cache module 103, and at least one preset value. The third selector outputs the preset value in the conventional convolution mode, and outputs the depth value in the depth perception convolution mode.

[0100] According to an embodiment of the present disclosure, taking the depth-aware convolution mode as an example, the third selector located in the first row outputs a first depth value to the absolute value calculation unit located in the first row and first column of the absolute value calculation matrix in the first system clock cycle when the depth coefficient calculation module 104 is turned on, and the absolute value calculation unit loads the depth value into reg, where reg is a register, and sends the first depth value to the absolute value calculation unit in the first row and second column, and in the second system clock cycle when the depth coefficient calculation module 104 is turned on, the third selector located in the first row outputs a second depth value to the absolute value calculation unit in the first row and first column, and the absolute value calculation unit repeats the operation of the first system clock cycle, and the absolute value calculation unit in the first row and second column sends the first depth value to the absolute value calculation unit in the first row and third column. In the third system clock cycle when the depth coefficient calculation module 104 is turned on, the third selector located in the first row outputs the third depth value to the absolute value calculation unit in the first row and the first column. At the same time, the cal in the absolute value calculation unit calculates the absolute value difference between the third depth value and the reference depth value, and the cal in the absolute value calculation unit in the first row and the second column calculates the absolute value difference between the second depth value and the reference depth value. The cal in the absolute value calculation unit in the first row and the third column calculates the absolute value difference between the first depth value and the reference depth value, and after obtaining the above-mentioned absolute value differences, they are stored in the output registers out_reg of each absolute value calculation unit.

[0101] According to an embodiment of the present disclosure, in the third system clock cycle or the fourth system clock cycle when the depth coefficient calculation module 104 is turned on, each absolute value calculation unit stores the absolute value difference in its own out_reg in the data stacking unit located in the first row, and the data stacking unit splices the three absolute value differences in sequence.

[0102] According to an embodiment of the present disclosure, the data stacking unit sends a cache selection signal to a first selector in an absolute value cache unit located in the first row, and the first selector determines a first cache subunit from at least two cache subunits according to the cache selection signal to cache stacked data.

[0103] According to an embodiment of the present disclosure, the asynchronous buffer unit located in the first row sends a data read signal to a second selector in the absolute value cache unit located in the first row, and the second selector determines a second cache sub-unit from at least two cache sub-units according to the data read signal, so that the asynchronous buffer unit reads stacked data from the second cache sub-unit.

[0104] According to an embodiment of the present disclosure, the lookup unit located in the first row reads the absolute value difference from the asynchronous buffer unit, finds the depth coefficient from a preset lookup table, and repeats the above process until the depth coefficients corresponding to at least one depth value are found respectively. After obtaining each depth coefficient, the depth coefficient is passed to the pre-calculation unit for pre-calculation.

[0105] According to an embodiment of the present disclosure, Figure 3 The numbers of the third selector, the absolute value calculation matrix, the data stacking unit, the absolute value cache unit, the asynchronous buffer unit, the lookup unit, and the pre-calculation unit in

[0106] According to an embodiment of the present disclosure, the absolute value calculation matrix is used to determine a reference depth value from at least one depth value, and determine the absolute value difference between the reference depth value and each of the at least one depth value. At least one data stacking unit, each data stacking unit corresponding to at least one target absolute value calculation unit in the absolute value calculation matrix, the data stacking unit is used to splice the absolute value differences output by the at least one target absolute value calculation unit in the same system clock cycle to obtain stacked data, wherein the at least one target absolute value calculation unit is located in the same row. At least one absolute value cache unit, respectively connected to the at least one data stacking unit, the absolute value cache unit is used to cache the stacked data.

[0107] According to an embodiment of the present disclosure, the absolute value calculation matrix includes J columns of absolute value calculation units. When in the i-th system clock cycle and i is not equal to N*J, in response to j = 1, the absolute value calculation unit in the j-th column of the absolute value calculation matrix is used to load the target depth value from at least one depth value sent by the depth value cache module 103, and is used to send the target depth value to the absolute value calculation unit in the (j + 1)-th column, wherein i is an integer greater than or equal to 1, and N and J are integers; in response to 1 < j < J, the absolute value calculation unit in the j-th column is used to load the target depth value sent by the absolute value calculation unit in the (j - 1)-th column, and is used to send the target depth value to the absolute value calculation unit in the (j + 1)-th column; in response to j = J, the absolute value calculation unit in the j-th column is used to load the target depth value sent by the absolute value calculation unit in the (j - 1)-th column.

[0108] According to an embodiment of the present disclosure, in the case where it is the i-th system clock cycle and i is equal to N*J, in response to j = 1, the absolute value calculation unit of the j-th column is used to load the target depth value sent by the depth value cache module 103, and is used to determine the target absolute value difference between the target depth value and the reference depth value, and is used to send the target depth value to the absolute value calculation unit of the j+1-th column, and send the target absolute value difference to the data stacking unit corresponding to the absolute value unit, where the reference depth value is the M-th target depth value sent by the depth value cache module 103, and M is an integer greater than or equal to 1; in response to 1 < j < J, the absolute value calculation unit of the j-th column is used to load the target depth value sent by the absolute value calculation unit of the j-1-th column, and is used to determine the target absolute value difference between the target depth value and the reference depth value, and is used to send the target depth value to the absolute value calculation unit of the j+1-th column, and send the target absolute value difference to the data stacking unit corresponding to the absolute value unit; in response to j = J, the absolute value calculation unit of the j-th column is used to load the target depth value sent by the absolute value calculation unit of the j-1-th column, and is used to determine the target absolute value difference between the target depth value and the reference depth value, and is used to send the target absolute value difference to the data stacking unit corresponding to the absolute value unit, and delete the target depth value.

[0109] According to an embodiment of the present disclosure, M in the M-th target depth value can be preset. For example, M = 1, that is, the first target depth value received by the absolute value calculation matrix is used as the reference depth value.

[0110] According to an embodiment of the present disclosure, after determining the reference depth value, the absolute value calculation unit for loading the reference depth value can send the reference depth value to each absolute value calculation unit in the absolute value calculation matrix.

[0111] According to an embodiment of the present disclosure, the absolute value cache unit includes a first selector, at least two cache subunits, and a second selector; the first selector is used to determine a first cache subunit from at least two cache subunits, where the first cache subunit is used to cache the written stacked data; the second selector is used to determine a second cache subunit from at least two absolute value cache units, and the second cache subunit is used to provide stacked data for the asynchronous buffer unit to read the stacked data.

[0112] According to an embodiment of the present disclosure, by making the absolute value cache unit include a first selector, at least two cache subunits, and a second selector, simultaneous reading and writing can be achieved, and the data reading and writing efficiency can be improved.

[0113] According to an embodiment of the present disclosure, the depth coefficient calculation module 104 further includes: an asynchronous buffer unit.

[0114] The asynchronous buffer unit is used to obtain at least one target stack data from the absolute value cache unit corresponding to the asynchronous buffer unit, and temporarily store the at least one target stack data according to a storage capacity threshold of the asynchronous buffer unit.

[0115] According to an embodiment of the present disclosure, the input feature map includes at least one input feature value; the pre-calculation module 105 includes a search unit and a pre-calculation unit; the search unit is used to obtain a target absolute value difference from at least one target stacked data stored in the asynchronous buffer unit based on a preset data amount, and is used to determine a depth coefficient corresponding to the target absolute value difference from a preset lookup table based on the target absolute value difference; the pre-calculation unit is used to multiply the depth coefficient with the corresponding input feature value to obtain a pre-calculation sub-result corresponding to the depth coefficient, and is used to obtain a pre-calculation result from the pre-calculation sub-result corresponding to each of the at least one depth coefficient, and is used to send the pre-calculation result to the systolic array 106.

[0116] According to an embodiment of the present disclosure, the input feature values ​​in the input feature map and the depth values ​​in the depth map have a one-to-one correspondence, that is, the input feature values ​​represent the pixel values ​​of each pixel, and the depth values ​​represent the depth values ​​of each pixel, that is, the pixel values ​​and depth coefficients of the same pixel can be pre-calculated. Since there is a corresponding relationship between the depth values ​​and the pixels, and there is also a corresponding relationship between the depth coefficients and the depth values, it is possible to determine the corresponding relationship between the depth coefficients and the pixels, and to determine the input feature values ​​corresponding to the depth coefficients.

[0117] According to an embodiment of the present disclosure, the pre-calculation unit may include a register If_reg for storing an input feature value, a register reg for storing a depth coefficient, and a multiplier. Figure 3 Among them, fmap_0, Ifmap_1 and Ifmap_2 represent the input feature value of the zeroth pixel, the input feature value of the first pixel and the input feature value of the second pixel, respectively; dx_0, dx_1 and dx_2 represent the depth coefficient of the zeroth pixel, the depth coefficient of the first pixel and the depth coefficient of the second pixel, respectively; pre_0, pre_1 and pre_2 represent the pre-calculated sub-result of the zeroth pixel, the pre-calculated sub-result of the first pixel and the pre-calculated sub-result of the second pixel, respectively.

[0118] According to an embodiment of the present disclosure, the pre-calculation between the depth coefficient and the input eigenvalue may be multiplication, after which a pre-calculation sub-result of the pixel point may be obtained, and there is a corresponding relationship between the pre-calculation sub-result and the depth coefficient or the input eigenvalue.

[0119] According to an embodiment of the present disclosure, the number of multipliers depends on the absolute value calculation matrix. Exemplarily, a 3*3 absolute value calculation matrix corresponds to pre-calculation using 3*1 multipliers.

[0120] According to an embodiment of the present disclosure, based on depth similarity, the depth-aware convolution formula is as shown in the following formula (2).

[0121] (2)

[0122] According to the embodiment of the present disclosure, it can be seen from formula (2) that pixels with similar depth to the convolution center have a greater impact on the output during the convolution process. Therefore, in the depth perception convolution mode, the pre-calculation module 105 can replace the original input feature map in the ordinary convolution after pre-processing the depth coefficient and the input feature map. Therefore, inputting the pre-calculation result into the systolic array 106 can realize the reuse of the systolic array 106, and whether it is the perception convolution mode or the conventional convolution mode, a systolic array 106 can be used to implement the convolution processing.

[0123] According to an embodiment of the present disclosure, after being processed by the pre-calculation module 105, the depth-aware convolution operation is converted into the output of a standard 2D convolution, as specifically shown in formula (3).

[0124] (3)

[0125] Among them, R represents the local grid around p0 in x, w is the convolution kernel, p0 and pn refer to the 0th pixel and nth pixel respectively.

[0126] According to the embodiments of the present disclosure, therefore, computation reuse can be achieved using the systolic array 106. The input feature map pre-computation module 105 uses a multiplier to implement the multiplication of the input feature map and the depth coefficient, which is equivalent to the input feature map of the input systolic array 106 after fine-tuning in the conventional convolution mode. The pre-computation result is loaded into the PE unit register of the systolic array 106 for subsequent convolution calculation.

[0127] According to an embodiment of the present disclosure, the depth value cache module 103 includes a third selector and a depth value cache unit; the depth value cache unit is used to cache at least one depth value; when the working mode is the conventional convolution mode, the third selector is used to output a depth value equal to a preset value; when the working mode is the depth-aware convolution mode, the selector is used to output in sequence from at least one depth value in the cache unit.

[0128] According to an embodiment of the present disclosure, the depth value cache unit may cache the depth value upon receiving a depth value loading enable signal sent by the depth value loading unit, and the depth value may be obtained from the external memory 111, which may be an external hard disk, disk or other device.

[0129] According to an embodiment of the present disclosure, under the control of a depth value determination enable signal and a depth value transmission enable signal, the depth value buffer module 103 determines a depth value corresponding to the current working mode and transmits it.

[0130] According to an embodiment of the present disclosure, the depth value cache unit in the depth value cache module 103 can be composed of a group of BRAMs (Block Random Access Memory), the size of which is determined by the size of at least one depth value, namely, the depth map. The depth map is used to provide real-time depth coefficients for depth-aware convolution. The size of the depth map to be cached is the same as the input feature map.

[0131] According to the embodiments of the present disclosure, the preset value is not limited and can be any value that has no effect on the conventional convolution mode, such as the preset value can be 1.

[0132] According to an embodiment of the present disclosure, specifically, when in the depth perception convolution mode, the configurable mode module controls the depth value cache module 103 to normally output the depth value to the depth coefficient calculation module 104; at the same time, the configurable mode module can also control the pre-calculation module 105 to output the pre-calculation result normally to the systolic array 106 to complete the depth perception convolution calculation.

[0133] According to an embodiment of the present disclosure, when the architecture works in the conventional convolution mode, the configurable mode module controls the depth value buffer module 103 to output a preset value such as 1, and sends it to the depth coefficient calculation module 104, so that the effective range of the enabling signal is the same as when calculating the depth perception convolution. At this time, the pre-calculation result output by the pre-calculation module 105 is the original value of the input feature map, and the systolic array 106 only loads the original feature map data and the convolution kernel for ordinary convolution calculation.

[0134] According to an embodiment of the present disclosure, the depth value output by the depth value buffer module 103 can be controlled so that the configurable depth-aware convolution operator hardware architecture can run in both conventional convolution mode and depth-aware convolution mode, and each module can be compatible with different modes.

[0135] According to an embodiment of the present disclosure, the configurable depth-aware convolution operator hardware architecture further includes: an input feature map cache module 108 and a weight cache module.

[0136] The input feature map cache module 108 is used to cache the input feature map, and to send the input feature map to the pre-calculation module 105 when receiving the input data sending enable signal sent by the task scheduling module 101.

[0137] The weight cache module is used to cache at least one weight value, and is used to send at least one weight value to the systolic array 106 when receiving a weight sending enable signal sent by the task scheduling module 101, so that the systolic array 106 performs convolution processing on the pre-calculated result based on the at least one weight value to obtain a convolution processing result.

[0138] According to an embodiment of the present disclosure, the input feature map cache module 108 may be composed of a group of BRAMs, and the number of BRAMs used depends on the size of the divided input feature map. The weight cache module may be composed of a group of BRAMs, and the number of BRAMs used depends on the size of the convolution window.

[0139] According to an embodiment of the present disclosure, the configurable depth-aware convolution operator hardware architecture also includes a cache data loading module 107; the cache data loading module 107 includes an input feature map loading unit, a depth value loading unit, and a weight loading unit.

[0140] An input feature map loading unit is used to load at least one depth value into the depth value cache module 103 upon receiving an input feature map loading enable signal sent by the task scheduling module 101; a depth value loading unit is used to load the input feature map into the input feature map cache module 108 upon receiving a depth value loading enable signal sent by the task scheduling module 101; a weight loading unit is used to load at least one weight value into the weight cache module upon receiving a weight loading enable signal sent by the task scheduling module 101.

[0141] According to an embodiment of the present disclosure, the configurable depth-aware convolution operator hardware architecture may further include an external memory 111 and an external memory controller 112 .

[0142] The external memory 111 is used to store at least one depth value, an input feature map, and at least one weight value.

[0143] The external memory controller 112 is used to control the external memory 111 so as to obtain at least one depth value, an input feature map, and at least one weight value from the external memory 111 .

[0144] According to an embodiment of the present disclosure, the external memory 111 may be a double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR).

[0145] According to an embodiment of the present disclosure, the cache data loading module 107 is mainly responsible for loading data from the DDR and writing it into the on-chip BRAM module for use by the convolution calculation unit or other processing units. The BRAM module may include: a depth value cache module 103, an input feature map cache module 108, and a weight value cache module 109.

[0146] According to an embodiment of the present disclosure, the cache data loading module 107 integrates the functions of the input feature map loading unit, the depth value loading unit and the weight loading unit, and uniformly manages the write enable signal of the BRAM to ensure the correct writing and caching of data.

[0147] According to the embodiments of the present disclosure, specifically, the input feature map loading module is mainly responsible for controlling the BRAM write enable, and writing the data from the DDR into the cache module BRAM on the input feature map. Specifically, taking the cache of three rows of input feature maps as an example, it is necessary to use two rows of BRAM in the input feature map cache module 108. When the first row of input feature values ​​arrives, the first row of BRAM write enable signal is pulled high, that is, set to 1, and a pixel value is written in each clock cycle. When the first row of BRAM is full, the second row of BRAM write enable signal is pulled high, and the first row of pixels is written into the second row of BRAM. At this time, the first row of BRAM keeps the write enable signal high, and continues to write the second row of input feature map pixels. When the second row of BRAM is full, the input and two BRAM data are read at the same time in the next clock cycle to obtain three rows of input feature maps.

[0148] According to an embodiment of the present disclosure, the depth map loading module is responsible for writing data from the DDR into the BRAM of the depth value cache module 103 and controlling the write enable signal of the BRAM. The weight loading module is mainly responsible for loading the weight value of the convolution kernel from the DDR and writing it into the BRAM of the weight cache module for use by the convolution calculation unit. That is, the cache data loading module 107 ensures that the weight data is correctly written by controlling the write enable signal of the BRAM.

[0149] According to an embodiment of the present disclosure, the configurable depth-aware convolution operator hardware architecture also includes: a post-module 110; the post-module 110 is used to perform pooling processing on the convolution processing result to obtain pooled data, and to perform mapping processing on the pooled data to obtain target output data.

[0150] According to an embodiment of the present disclosure, the post-module 110 may include any unit that can be added after convolution, such as a pooling unit, a mapping unit, and a normalization unit, and an activation function processing unit.

[0151] According to an embodiment of the present disclosure, the target output data may be used in the next step or stored in a memory such as a DDR.

[0152] According to an embodiment of the present disclosure, the pooling unit may perform pooling processing on the convolution processing result. The mapping unit may be placed after the pooling unit for a fully connected neural network, and map the pooled convolution processing result to the required target output data.

[0153] Figure 4 The figure schematically shows a task processing flow chart of a configurable depth-aware convolution operator hardware architecture according to an embodiment of the present disclosure.

[0154] like Figure 4 As shown, the task processing flow chart includes operations S401 to S407.

[0155] In operation S401 , the task scheduling module receives working mode information.

[0156] In operation S402, the task scheduling module determines whether the working mode is the depth-aware convolution mode. If the working mode is determined to be the conventional convolution mode, operation S403 is performed; if the working mode is determined to be the depth-aware convolution mode, operation S405 is performed.

[0157] In operation S403, the task scheduling module sends a mode enable signal to the configurable module, sends an input feature map loading enable signal to the input feature map loading unit, and sends a weight loading enable signal to the weight loading unit.

[0158] According to an embodiment of the present disclosure, a mode enable signal is sent to a configurable module so that the configurable module sends a control to a depth value cache module to determine a depth value corresponding to the working mode, an input feature map loading enable signal is sent to an input feature map loading unit so that the input feature map loading unit controls the input feature map cache module to cache input feature values, and a weight loading enable signal is sent to a weight loading unit so that the weight loading unit controls the weight cache unit to cache weight values.

[0159] In operation S404, when the task scheduling module determines that the input feature value cached by the feature map cache module meets the first cache condition, the task scheduling module sends an input data sending enable signal so that the input feature map cache module sends the input feature map to the pre-calculation module; when the task scheduling module determines that the weight value cached by the weight loading unit meets the second cache condition, the task scheduling module sends a weight value sending enable signal weight value to the systolic array 106; when the task scheduling module determines that the depth value cache module has determined the depth value, the task scheduling module sends a depth value sending enable signal to the depth value cache module so that the depth value cache module sends the depth value to the depth coefficient calculation module.

[0160] In operation S405, the task scheduling module sends a depth value loading enable signal to the depth value loading unit, sends a mode enable signal to the configurable module, sends an input feature map loading enable signal to the input feature map loading unit, and sends a weight loading enable signal to the weight loading unit.

[0161] According to an embodiment of the present disclosure, the task scheduling module sends a depth value loading enable signal to the depth value loading unit so that the depth value loading unit controls the depth value cache module to cache the depth value, sends a mode enable signal to the configurable module so that the configurable module controls the depth value cache module to determine the depth value corresponding to the working mode, sends an input feature map loading enable signal to the input feature map loading unit so that the input feature map loading unit controls the input feature map cache module to cache the input feature value, and sends a weight loading enable signal to the weight loading unit so that the weight loading unit controls the weight cache unit to cache the weight value.

[0162] In operation S406, when the task scheduling module determines that the input feature value cached by the input feature map cache module meets the first cache condition, the task scheduling module sends an input data sending enable signal so that the input feature map cache module sends the input feature map to the pre-calculation module; when the task scheduling module determines that the weight value cached by the weight value cache unit meets the second cache condition, the task scheduling module sends a weight value sending enable signal weight value to the systolic array 106; when the task scheduling module determines that the weight value cached by the depth value cache module meets the third cache condition and the depth value cache module has determined the depth value, the task scheduling module sends a depth value sending enable signal to the depth value cache module so that the depth value cache module sends the depth value to the depth coefficient calculation module.

[0163] According to an embodiment of the present disclosure, the first cache condition is determined based on the BRAM in the input feature map cache module 108, such as: when it is determined that each row of BRAM stores the input feature value, it is considered that the first cache condition has been met. The second cache condition is determined based on the BRAM in the weight value cache unit. The third cache condition is determined based on the BRAM in the depth value cache module 103.

[0164] In operation S407 , the depth coefficient calculation module, the pre-calculation module, and the systolic array 106 are started in sequence for processing.

[0165] According to an embodiment of the present disclosure, the task scheduling module 101 can send a corresponding module start enable signal to start the module based on the data received by the depth coefficient calculation module 104, the pre-calculation module 105 and the systolic array 106, respectively, and when it is determined that the module has received the data required for calculation. For example, when the depth coefficient calculation module 104 receives a depth value, the depth coefficient calculation module 104 can be started to execute the execution logic of the module.

[0166] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0167] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0168] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A configurable depth-aware convolution operator hardware architecture, characterized in that: Including: A task scheduling module, configured to generate a data enable signal and control the execution order of convolution tasks based on the data enable signal, where the data enable signal includes a mode enable signal and a depth value transmission enable signal; A configurable module, configured to determine a working mode based on the mode enable signal and send a depth value determination enable signal corresponding to the working mode to a depth value cache module, where the working mode includes a conventional convolution mode or a depth-aware convolution mode; The depth value cache module is configured to determine at least one depth value corresponding to the working mode when receiving the depth value determination enable signal, and send at least one depth value corresponding to the working mode to a depth coefficient calculation module respectively when receiving the depth value transmission enable signal; The depth coefficient calculation module is configured to obtain depth coefficients corresponding to at least one of the depth values based on at least one of the depth values and send the depth coefficients corresponding to at least one of the depth values to a pre-calculation module; The pre-calculation module is configured to perform pre-calculation on the depth coefficients corresponding to at least one of the depth values and an input feature map to obtain a pre-calculation result and send the pre-calculation result to a systolic array; The systolic array is configured to perform convolution processing on the pre-calculation result to obtain a convolution processing result.

2. The configurable depth-aware convolution operator hardware architecture according to claim 1, characterized in that: The depth coefficient calculation module includes: an absolute value calculation matrix, at least one data stacking unit, and at least one absolute value cache unit; The absolute value calculation matrix is configured to determine a reference depth value from at least one of the depth values and determine the absolute value differences between the reference depth value and each of at least one of the depth values; At least one of the data stacking units, each data stacking unit corresponding to at least one target absolute value calculation unit in the absolute value calculation matrix, the data stacking unit being configured to splice the absolute value differences output by at least one of the target absolute value calculation units in the same system clock cycle to obtain stacked data, where at least one of the target absolute value calculation units is in the same row; At least one of the absolute value cache units, respectively connected to at least one of the data stacking units, the absolute value cache unit being configured to cache the stacked data.

3. The configurable depth-aware convolution operator hardware architecture according to claim 2, characterized in that: The absolute value calculation matrix includes J columns of absolute value calculation units; When in the i-th system clock cycle and i is not equal to N*J, In response to j = 1, the absolute value calculation unit in the j-th column of the absolute value calculation matrix is configured to load a target depth value from at least one of the depth values sent by the depth value cache module and send the target depth value to the absolute value calculation unit in the j + 1-th column, where i is an integer greater than or equal to 1, and N and J are integers; In response to 1 < j < J, the absolute value calculation unit in the j-th column is configured to load the target depth value sent by the absolute value calculation unit in the j - 1-th column and send the target depth value to the absolute value calculation unit in the j + 1-th column; In response to j = J, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the absolute value calculation unit of the (j - 1)-th column; In the case where it is at the i-th system clock cycle and i is equal to N * J, In response to j = 1, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the depth value cache module, and is configured to determine the target absolute value difference between the target depth value and the reference depth value, and is configured to send the target depth value to the absolute value calculation unit of the (j + 1)-th column, and send the target absolute value difference to the data stacking unit corresponding to the absolute value unit, wherein the reference depth value is the M-th target depth value sent by the depth value cache module, and M is an integer greater than or equal to 1; In response to 1 < j < J, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the absolute value calculation unit of the (j - 1)-th column, and is configured to determine the target absolute value difference between the target depth value and the reference depth value, and is configured to send the target depth value to the absolute value calculation unit of the (j + 1)-th column, and send the target absolute value difference to the data stacking unit corresponding to the absolute value unit; In response to j = J, the absolute value calculation unit of the j-th column is configured to load the target depth value sent by the absolute value calculation unit of the (j - 1)-th column, and is configured to determine the target absolute value difference between the target depth value and the reference depth value, and is configured to send the target absolute value difference to the data stacking unit corresponding to the absolute value unit, and delete the target depth value.

4. The configurable depth-aware convolution operator hardware architecture according to claim 2, wherein: The absolute value cache unit includes a first selector, at least two cache subunits, and a second selector; The first selector is configured to determine a first cache subunit from at least two cache subunits, wherein the first cache subunit is configured to cache the written stacking data; The second selector is configured to determine a second cache subunit from at least two absolute value cache units, and the second cache subunit is configured to provide the stacking data so that the asynchronous buffer unit reads the stacking data.

5. The configurable depth-aware convolution operator hardware architecture according to claim 4, characterized in that: The depth coefficient calculation module further includes: the asynchronous buffer unit; The asynchronous buffer unit is configured to obtain at least one target stacking data from the absolute value cache unit corresponding to the asynchronous buffer unit, and temporarily store at least one of the target stacking data according to the storage capacity threshold of the asynchronous buffer unit.

6. The configurable depth-aware convolution operator hardware architecture according to claim 5, characterized in that: The input feature map includes at least one input feature value; the pre-computation module includes a lookup unit and a pre-computation unit; The lookup unit is configured to obtain the target absolute value difference from at least one of the target stacking data stored in the asynchronous buffer unit based on a preset data volume, and is configured to determine the depth coefficient corresponding to the target absolute value difference from a preset lookup table based on the target absolute value difference; The pre-computation unit is configured to multiply the depth coefficient by the corresponding input feature value to obtain a pre-computation sub-result corresponding to the depth coefficient, and is configured to obtain a pre-computation result from the pre-computation sub-results corresponding to at least one depth coefficient respectively, and is configured to send the pre-computation result to the systolic array.

7. The configurable depth-aware convolution operator hardware architecture according to claim 1, characterized in that: The depth value cache module includes a third selector and a depth value cache unit; The depth value cache unit is used to cache at least one of the depth values; When the working mode is the conventional convolution mode, the third selector is used to output a depth value equal to a preset value; When the operating mode is the depth-aware convolution mode, the third selector is used to output at least one of the depth values ​​in the depth value cache unit in sequence.

8. The configurable depth-aware convolution operator hardware architecture of claim 1, wherein: The configurable depth-aware convolution operator hardware architecture also includes: an input feature map cache module and a weight cache module; The input feature map cache module is used to cache the input feature map, and is used to send the input feature map to the pre-calculation module when receiving the input data sending enable signal sent by the task scheduling module; The weight cache module is used to cache at least one weight value, and is used to send at least one weight value to the systolic array when receiving a weight sending enable signal sent by the task scheduling module, so that the systolic array performs convolution processing on the pre-calculated result based on the at least one weight value to obtain a convolution processing result.

9. The configurable depth-aware convolution operator hardware architecture of claim 8, wherein: The configurable depth-aware convolution operator hardware architecture further includes a cache data loading module; the cache data loading module includes an input feature map loading unit, a depth value loading unit, and a weight loading unit; The input feature map loading unit is used to load at least one depth value into the depth value cache module when receiving the input feature map loading enable signal sent by the task scheduling module; The depth value loading unit is used to load the input feature map into the input feature map cache module when receiving the depth value loading enable signal sent by the task scheduling module; The weight loading unit is used to load at least one weight value into the weight cache module when receiving a weight loading enable signal sent by the task scheduling module.

10. The configurable depth-aware convolution operator hardware architecture according to claim 1, wherein: Also includes: Post-processing module; The post-processing module is used to perform pooling processing on the convolution processing results to obtain pooled data, and to perform mapping processing on the pooled data to obtain target output data.

Images