Signal processing device, imaging device, and signal processing method

Abstract

A signal processing apparatus is configured to include: a multiply-accumulate unit positioned in the form of a one-dimensional or two-dimensional array and capable of performing multiply-accumulate operations in a neural network; a threshold evaluation processing unit for evaluating whether input data for calculations performed by the multiply-accumulate unit is less than a predetermined threshold; and an avoidance processing unit for avoiding multiply-accumulate operations on the input data when the input data is less than the predetermined threshold.

Description

Technical Field

[0001] This technology relates to a signal processing apparatus, an imaging apparatus, and a signal processing method for performing multiplication and accumulation operations. Background Technology

[0002] There are cases where processing related to deep neural networks (DNNs), such as image recognition processing of objects, is performed on images captured by imaging devices (such as cameras). In such processing related to DNNs (e.g., image recognition processing, etc.), many multiplication and accumulation operations are required.

[0003] In multiplication-accumulation operations, two types of input data are used, such as image data and weight data. Both types of input data may include many zero values, and in this case, there is a problem of performing useless operations and not making efficient use of memory.

[0004] In response to such problems, for example, Patent Document 1 discloses a technique for generating an index that includes one or more memory address locations having input data (input activation values) as non-zero values. It describes how input data can be compressed and computational efficiency improved by storing only non-zero input data in memory.

[0005] Reference List

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Publication No. 2020-500365 Summary of the Invention

[0008] The problem to be solved by the present invention

[0009] Meanwhile, in the multiplication and accumulation operations performed in image recognition processing, there are cases where the input data has a low bit length and cases where the input data includes multiple non-zero values.

[0010] In this case, if an index including the memory address location is generated and stored in memory, there is a possibility that the memory utilization efficiency or computational efficiency will be significantly reduced.

[0011] In view of the above, this technology is proposed, the purpose of which is to improve the computational efficiency of multiplication and accumulation operations.

[0012] Solution to the problem

[0013] The signal processing apparatus according to the present technology includes: a multiplication-accumulation operation unit arranged in a one-dimensional or two-dimensional array and capable of performing multiplication-accumulation operations in a neural network; a threshold determination processing unit for determining whether the input data for operation by the multiplication-accumulation operation unit is less than a predetermined threshold; and an avoidance processing unit for avoiding multiplication-accumulation operation processing on the input data if the input data is less than the predetermined threshold.

[0014] Input data that is less than a predetermined threshold is, for example, input data with a value of zero or input data that is close to zero.

[0015] In the above signal processing device, the input data may include a first type of input data and a second type of input data. The threshold determination processing unit can perform the determination of the first type of input data, and if the first type of input data is less than the predetermined threshold, the avoidance processing unit can avoid the multiplication and accumulation operation processing of the first type of input data.

[0016] The multiplication-accumulation unit multiplies the first type of input data with the second type of input data. That is, if either the first type of input data or the second type of input data is zero, the product is also zero. This configuration avoids multiplication-accumulation operations when the first type of input data is zero.

[0017] In the above-mentioned signal processing device, the second type of input data can be weighted data, which is information about the weights of multiplying the first type of input data.

[0018] Weight data, for example, are the coefficients of filters applied to a predetermined range of image data in a convolutional neural network (CNN). It is difficult to consider filters where all filter coefficients are zero.

[0019] A threshold determination processing unit from the aforementioned signal processing device can be provided for each of the multiple multiplication and accumulation operation units.

[0020] Determine whether each of the multiple input data input to multiple multiplication and accumulation units is less than a predetermined threshold, for example, whether the input data is zero.

[0021] The avoidance processing unit in the above signal processing device can change the input data to the multiplication and accumulation operation unit when the input data is less than a predetermined threshold, in order to avoid performing multiplication and accumulation operation processing on the input data that is less than the predetermined threshold.

[0022] Therefore, input data that is equal to or greater than a predetermined threshold is input to the multiplication and accumulation unit.

[0023] The aforementioned signal processing apparatus may include: a multiplication-accumulation operation control unit, which manages the input data and output data of the multiplication-accumulation operation processing, wherein the avoidance processing unit may notify the multiplication-accumulation operation control unit of information specifying the input data that has been avoided from the multiplication-accumulation operation processing.

[0024] Therefore, the multiplication and accumulation operation control unit can grasp the correspondence between the input data used in the multiplication and accumulation operation and the result of the multiplication and accumulation operation.

[0025] An avoidance processing unit in the above-mentioned signal processing device can be set for each multiplication and accumulation operation unit.

[0026] By providing avoidance processing units for each multiplication-accumulation operation unit, the processing load of the deterministic process performed by a single avoidance processing unit can be reduced. In this deterministic process, it is determined whether the input data is less than a predetermined threshold, for example, whether the input data is zero.

[0027] The avoidance processing unit in the above signal processing device can avoid performing multiplication and accumulation operations on input data that is less than a predetermined threshold, and output a zero value as the processing result of the multiplication and accumulation operation.

[0028] For example, it is obvious that the result of the operation will be zero when the input data is zero, so we should avoid forcing the output data to zero after the multiplication and accumulation operation.

[0029] In the above-described signal processing device, the input data may include a first type of input data and a second type of input data. If the first type of input data is less than a first threshold, the avoidance processing unit may change the first type of input data input to the multiplication and accumulation unit and notify the multiplication and accumulation control unit of information for specifying the changed first type of input data.

[0030] Therefore, for only one of the first type of input data and the second type of input data, a comparison process with a predetermined threshold can be performed, such as determining whether the input data is a zero value.

[0031] In the avoidance processing unit of the above-mentioned signal processing device, when the second type of input data is less than the second threshold, the avoidance processing unit can change the second type of input data input to the multiplication and accumulation operation unit and change the first type of input data corresponding to the changed second type of input data, and notify the multiplication and accumulation operation control unit of information for specifying the changed first type of input data and the changed second type of input data.

[0032] In multiplication and accumulation operations, the corresponding data is the number to be multiplied. During multiplication, if any of the numbers to be multiplied is zero, the result is zero, regardless of the value of the numbers being multiplied. To omit this multiplication process, the number of multiplications that result in zero values ​​(second-type input data) and the corresponding number of multiplications are omitted.

[0033] The multiplication and accumulation control unit in the aforementioned signal processing device can manage the multiplication and accumulation results of the first type of input data and the second type of input data, and compensate for zero values ​​for the avoided multiplication and accumulation results.

[0034] The multiply-accumulate operation to be avoided (i.e., the multiply-accumulate operation to be skipped) can be specified by receiving information for specifying the corresponding first type of input data and second type of input data.

[0035] An imaging apparatus according to the present technology includes: a pixel array unit, wherein photoelectric conversion elements are arranged in a one-dimensional or two-dimensional array; a signal processing unit, to which input data based on the output signal of the pixel array unit is input; wherein the signal processing unit includes a multiplication-accumulation operation unit arranged in a one-dimensional or two-dimensional array and capable of performing multiplication-accumulation operations in a neural network; a threshold determination processing unit that determines whether the input data for operation by the multiplication-accumulation operation unit is less than a predetermined threshold; and an avoidance processing unit that avoids multiplication-accumulation operation processing on the input data if the input data is less than the predetermined threshold.

[0036] Due to issues such as battery problems, the signal processing unit, including in the imaging device, needs to be power-efficient.

[0037] In the aforementioned imaging device, the pixel array unit and the signal processing unit can be integrated into one unit.

[0038] By forming them as a whole, imaging devices can be miniaturized.

[0039] In the signal processing unit of the aforementioned imaging device, the feature data extracted based on the output signal of the pixel array unit can be input to the signal processing unit as input data.

[0040] Feature data typically includes data with zero values ​​or values ​​less than a predetermined threshold.

[0041] The signal processing method according to this technology is a signal processing method for processing performed by a signal processing device, the processing including: determining whether input data for a multiplication-accumulation operation in a neural network is less than a predetermined threshold; and avoiding multiplication-accumulation operation processing on the input data if the input data is less than the predetermined threshold.

[0042] Even using this signal processing method, similar operation and effects to the signal processing device according to the present technology can be obtained. Attached Figure Description

[0043] Figure 1 This is a diagram illustrating an example configuration of an imaging apparatus according to an embodiment of the present technology.

[0044] Figure 2 This is a diagram showing an example of the internal configuration of a sensor unit.

[0045] Figure 3 This is a diagram showing an example configuration of a signal processing unit.

[0046] Figure 4 This is a diagram illustrating an example of processing target data (pixel data) and target regions.

[0047] Figure 5 This is a diagram illustrating an example of a filter applied to the target region.

[0048] Figure 6 It is a diagram used to describe the multiplication and accumulation operation performed in a MAC.

[0049] Figure 7 This is a diagram showing an example 1 of the configuration of a signal processing unit.

[0050] Figure 8 It is used to describe and Figure 9 A diagram showing the processing of pixel data as input data in the configuration example 1 of the signal processing unit is shown, and the diagram shows the state before the replacement.

[0051] Figure 9 This is a diagram showing the state after the pixel data used as input data has been replaced.

[0052] Figure 10 This is a diagram showing an example 2 of the configuration of the signal processing unit.

[0053] Figure 11 This is a diagram illustrating an example of a filter in configuration example 2 of the signal processing unit.

[0054] Figure 12 This is a diagram illustrating an example of the target area in Example 2 of the configuration of the signal processing unit.

[0055] Figure 13 It is used to describe and Figure 14 and Figure 15 A diagram showing the replacement of the weight data as input data in the configuration example 2 of the signal processing unit, and the diagram showing the state before the replacement.

[0056] Figure 14 This is a diagram showing the weight data that will be replaced.

[0057] Figure 15 This is a diagram showing the state after the weight data used as input data has been replaced.

[0058] Figure 16 This is a diagram illustrating an example of the filter and target region in configuration example 3 of the signal processing unit.

[0059] Figure 17 This is a diagram illustrating a configuration example 3 of the signal processing unit.

[0060] Figure 18 It is used to describe and Figure 19 A diagram showing the processing of pixel data as input data in the configuration example 3 of the signal processing unit is shown, and the diagram shows the state before the replacement.

[0061] Figure 19 This is a diagram showing the state after replacing the pixel data that was used as input data.

[0062] Figure 20 This is a diagram illustrating a configuration example 4 of the signal processing unit.

[0063] Figure 21 This is a diagram illustrating a configuration example of the MAC in configuration example 4 of the signal processing unit.

[0064] Figure 22 This is a flowchart illustrating a first processing example.

[0065] Figure 23 This is a flowchart illustrating a second processing example.

[0066] Figure 24 This is a flowchart illustrating a second processing example.

[0067] Figure 25 This is a flowchart illustrating a third processing example.

[0068] Figure 26 This is a diagram illustrating a configuration example of the MAC in the second variation.

[0069] Figure 27 This is a diagram illustrating an example where the signal processing unit is located in a control unit outside the sensor unit.

[0070] Figure 28 This is a diagram illustrating an example where the signal processing unit is located outside the sensor unit and outside the control unit.

[0071] Figure 29 This is a diagram illustrating an example where the signal processing unit is located outside the imaging device. Detailed Implementation

[0072] In the following description, embodiments according to the present technology will be described in the following order with reference to the accompanying drawings.

[0073] <1. Configuration of Imaging Device>

[0074] <2. Specific Configuration Example of Signal Processing Unit>

[0075] <2-1. Configuration Example 1>

[0076] <2-2. Configuration Example 2>

[0077] <2-3. Configuration Example 3>

[0078] <2-4. Configuration Example 4>

[0079] <3. Flowchart>

[0080] <3-1. First Processing Example>

[0081] <3-2. Second Processing Example>

[0082] <3-3. Third Processing Example>

[0083] <4. Variations>

[0084] <4-1. First Variation>

[0085] <4-2. Second Variation>

[0086] <4-3. Variations of the sensor unit>

[0087] <4-4 Other modifications>

[0088] <5. Summary>

[0089] <6. This technology>

[0090] <1. Configuration of Imaging Device>

[0091] The signal processing apparatus of this technology is capable of performing various operations related to image recognition processing using deep neural networks (DNNs). In the following example, a signal processing apparatus will be described that performs multiplication-accumulation operations as image recognition processing using a convolutional neural network (CNN), which is a type of DNN.

[0092] Furthermore, various usage modes of the signal processing device are conceivable. The following examples will describe instances of providing and using a signal processing device in an imaging apparatus.

[0093] like Figure 1As shown, the imaging device 1 includes an imaging lens 2, a sensor unit 3, a control unit 4, and a recording unit 5.

[0094] Assume various modes of imaging device 1, such as cameras mounted on industrial robots, vehicle-mounted cameras, surveillance cameras, etc.

[0095] Imaging lens 2 converges the incident light and guides the light to sensor unit 3. Imaging lens 2 may include multiple lenses.

[0096] The sensor unit 3 includes multiple light receiving elements and outputs a signal obtained through photoelectric conversion.

[0097] The control unit 4 executes the shutter speed control of the sensor unit 3, commands for various types of signal processing including each unit in the imaging device 1, imaging and recording operations according to user operations, reproduction operations of recorded image files, drive control of the imaging lens 2 (e.g., zoom control, focus control, aperture control, etc.), user interface control, etc.

[0098] The recording unit 5 stores information, etc., that is processed by the control unit 4. Examples of recording units 5 include, for instance, read-only memory (ROM), random access memory (RAM), flash memory, etc.

[0099] The recording unit 5 may be a storage area built into the microcomputer chip that serves as the control unit 4, or it may include a separate storage chip.

[0100] The control unit 4 controls the entire imaging device 1 by executing programs stored in the ROM, flash memory, etc. of the recording unit 5.

[0101] Reference Figure 2 The sensor unit 3 is described in detail. The sensor unit 3 includes a pixel array unit 11, an arbitrator 12, a readout unit 13, a signal processing unit 14, and an output unit 15, which are used as a so-called dynamic vision sensor (DVS).

[0102] Note that sensor unit 3 is not limited to DVS and can be configured as a variety of image sensors.

[0103] In the pixel array unit 11, pixels 16, each including a photoelectric conversion element, are arranged in a two-dimensional array in the row direction (horizontal direction) and column direction (vertical direction).

[0104] Each pixel 16 detects the presence or absence of an event by whether the change in the amount of received light exceeds a predetermined threshold, and outputs a request to the arbitrator 12 when an event occurs.

[0105] Arbitrator 12 arbitrates requests from each pixel 16 and controls the reading operation of reading unit 13.

[0106] Based on the control of the arbitrator 12, the reading unit 13 performs a reading operation on each pixel 16 of the pixel array unit 11.

[0107] Each pixel 16 outputs a signal based on the difference between the reference level and the current level of the light-received signal, according to the readout action of the readout unit 13.

[0108] The signal read from each pixel 16 is stored in memory as a differential signal.

[0109] Furthermore, pixel 16 resets the reference level to the level of the current light-receiving signal based on the output of the differential signal. Therefore, the change in the amount of light received relative to the reference level can be detected again.

[0110] The differential signal reading and reference level reset are only performed when the change in the amount of light received exceeds a predetermined threshold.

[0111] The signal processing unit 14 performs various signal processing (preprocessing, etc.) and DNN image recognition processing on the image data input from the reading unit 13 as feature data. In the following description, image recognition processing as a DNN-type CNN will be described as an example.

[0112] Specifically, as image recognition processing, for example, operations related to convolution processing through convolutional layers, max-gathering processing through pooling layers, and classification processing through fully connected layers and output layers can be performed. In the following description, the multiple accumulation operations such as convolution processing performed in the signal processing unit 14 will be described as an example of image recognition processing.

[0113] The output unit 15 outputs the classification results of the CNN to the control unit 4 in the subsequent stage based on a predetermined interface standard (e.g., Mobile Industry Processor Interface (MIPI)).

[0114] Control unit 4 receives classification results via CNN and uses the classification results for various types of processing.

[0115] It should be noted that when the signal processing unit 14 performs only a portion of the various processes related to CNN, the processing results in the signal processing unit 14, i.e., the intermediate processing results in CNN, are output from the output unit 15.

[0116] Reference Figure 3 A configuration example of the signal processing unit 14 is described.

[0117] The signal processing unit 14 includes a MAC array unit 17, a signal processing control unit 18, and a memory unit 19 to perform multiplication and accumulation operations.

[0118] The MAC array unit 17 includes multiply-accumulate (MAC) units arranged in a two-dimensional array along the row (horizontal) and column (vertical) directions. Note that the multiply-accumulate units can also be arranged in a one-dimensional array along either the row or column direction.

[0119] The multiplication and accumulation unit is also called MAC 20.

[0120] In each MAC 20, circuitry is formed for performing multiplication and addition operations on data input from memory unit 19.

[0121] For example, the input data to a MAC 20 is either the data of a single pixel of the image data output from the pixel array unit 11, or weight data to be multiplied by the data of a single pixel. The weight data are the filter coefficients of the filter applied to the image data.

[0122] Note that the image data input to MAC 20 can be not only the image data output from pixel array unit 11, but also the image data output from another convolutional layer or pooling layer. In the following description, this image data is referred to as "processing target data".

[0123] An example of the operation performed by MAC 20 will be described by using the processed object data represented by binary values ​​(0 and 1) and a filter with two pixels, vertical and horizontal, to be applied to the processed object data.

[0124] Figure 4 This is a diagram showing the target region AR1, which is the target area processed as a filter. In the four pixels of target region AR1, the values ​​of the top-left pixel data a11 and the top-right pixel data a12 are both "1", while the values ​​of the bottom-left pixel data a21 and the bottom-right pixel data a22 are both "0".

[0125] Figure 5 This is a diagram showing the filter F1 applied to the target region AR1. The coefficients of filter F1 are the weight data w11, w12, w21, and w22.

[0126] In filter F1, the values ​​of the top-left weight data w11 and the bottom-right weight data w22 are "1", and the values ​​of the top-right weight data w12 and the bottom-left weight data w21 are "0".

[0127] In this case, during convolution processing (see...) Figure 6 In ), the operation of the following expression (1) is performed.

[0128] The expression (1) is: a11×w11+a12×w12+a21×w21+a22×w22...

[0129] The operation of expression (1) can be performed using four MAC 20s.

[0130] For example, pixel data a11 and weight data w11 are input to MAC 20a. Then, in MAC 20a, the multiplication of pixel data a11 and weight data w11 is performed, and the multiplication result is output as output OP1.

[0131] Not only pixel data a12 and coefficient w12, but also output OP1 is input to MAC 20b. MAC 20b performs multiplication of pixel data a12 and coefficient w12, and further performs addition of the multiplication result and output OP1. The sum is output as output OP2.

[0132] Pixel data a21, weight data w21, and output OP2 are input to MAC 20c. MAC 20c performs a multiplication of pixel data a21 and weight data w21, and then performs an addition of the multiplication result and output OP2. The sum is output as output OP3.

[0133] Pixel data a22, weight data w22, and output OP3 are input to MAC 20d. MAC 20d performs multiplication of pixel data a22 and weight data w22, and then performs addition of the multiplication result and output OP3. The sum is output as output OP4.

[0134] Therefore, the result of the operation of expression (1) from MAC 20d is used as output OP4.

[0135] It should be noted that Figure 6 The examples shown are illustrative, and for example, MACs 20a, 20b, 20c, and 20d can be controlled to perform only multiplication operations. In this case, the process of adding outputs OP1, OP2, OP3, and OP4 can be performed in MAC 20 other than MACs 20a, 20b, 20c, and 20d. Of course, MAC 20d can be configured to perform the process of adding outputs OP1, OP2, and OP3 to the multiplication result, so that output OP4 becomes the result of expression (1).

[0136] Description returned Figure 3 .

[0137] The signal processing control unit 18 performs the process of reading the processing target data (pixel data) and filter coefficients (weight data) stored in the memory unit 19 and inputting the data to each MAC 20 of the MAC array unit 17. Furthermore, the signal processing control unit 18 has a function to prevent the calculation result from becoming zero. This will be described in detail later.

[0138] The signal processing control unit 18 performs processing that stores the calculation results of the MAC array unit 17 in the memory unit 19. Furthermore, it performs processing on the calculation results outside the signal processing control unit 18.

[0139] exist Figure 1 , Figure 2 and Figure 3 The imaging apparatus 1 shown is an example including an image sensor, wherein the pixel array unit 11 and the signal processing unit 14 are integrally formed. For example, this is an example in which the pixel array unit 11, etc., is arranged on the front surface and the GPU, DSP, etc., are formed as the signal processing unit 14 on the rear surface.

[0140] However, the image sensor may not include the signal processing unit 14. That is, the image sensor and the signal processing unit 14 can be provided separately.

[0141] <2. Specific Configuration Example of Signal Processing Unit>

[0142] A specific configuration example of the signal processing unit 14 will be described with reference to the accompanying drawings.

[0143] <2-1. Configuration Example 1>

[0144] exist Figure 7 The specific configuration of the signal processing unit 14A in Configuration Example 1 is shown in the figure.

[0145] In the signal processing unit 14A of Configuration Example 1, an avoidance processing unit 21 is provided in either of the two data inputs to the multiplication circuit of the MAC 20, specifically the aforementioned pixel data and weight data (filter coefficients). Furthermore, one avoidance processing unit 21 is provided for each of the plurality of MACs 20. Figure 7 In the example shown, an avoidance processing unit 21 is provided for a MAC array unit 17 that includes multiple MACs 20.

[0146] like Figure 7 As shown, the signal processing unit 14A includes an avoidance processing unit 21, a first memory 22, a second memory 23, a third memory 24, a multiplication and accumulation operation control unit 25, a first local memory 26, a second local memory 27, and a plurality of MACs 20 arranged in a two-dimensional array and constituting a MAC array unit 17.

[0147] The avoidance processing unit 21 and the multiplication and accumulation operation control unit 25 are Figure 3 The signal processing control unit 18 shown.

[0148] Furthermore, the first memory 22, the second memory 23, and the third memory 24 are Figure 3 The memory unit 19 shown. The first memory 22, the second memory 23, and the third memory 24 can be configured as physically different memories, or they can be configured as different regions of a single memory.

[0149] The first memory 22 stores image data as processing target data. The second memory 23 stores weight data. The third memory 24 stores the calculation results. The calculation results stored in the third memory 24 can be output from the signal processing unit 14, or can be output to the first memory 22 as processing target data input to the MAC array unit 17. Note that the calculation results stored in the third memory 24 can be input to the MAC array unit 17 from the third memory 24 without passing through the first memory 22.

[0150] The processing unit 21 avoids reading the processing target data from the first memory 22 and inputting the processing target data to each MAC 20 of the MAC array section 17 via the first local memory 26.

[0151] The weight data stored in the second memory 23 is temporarily stored in the second local memory 27 and then input to each MAC 20 of the MAC array unit 17.

[0152] In each MAC 20, the pixel data of one pixel is multiplied by the weight data in the input processing target data.

[0153] Here, depending on the target data being processed from the input, the multiplication and accumulation operations in MAC 20 may be wasted. For example, in Figure 4 , Figure 5 and Figure 6 In the example shown, when all pixel data a11, a12, a21 and a22 are zero, the result of expression (1) always becomes zero, regardless of the values ​​of weight data w11, w12, w21 and w22, so there is no need to perform multiplication and accumulation.

[0154] The avoidance processing unit 21 performs processing to avoid the unwanted action.

[0155] This will refer to Figure 8 and Figure 9 Detailed description.

[0156] Figure 8 yes Figure 7 An excerpt of the MAC array unit 17 shown is presented. Specifically, eight MACs 20-1, MAC 20-2, MAC 20-3, MAC 20-4, MAC 20-5, MAC 20-6, MAC 20-7, and MAC 20-8 are shown among the plurality of MACs 20.

[0157] The four MAC 20s in MAC 20-1, 20-2, 20-3 and 20-4 are multiplication-accumulation units that perform convolution processing on the target region AR1 of the object data to which a filter is applied.

[0158] The four MAC 20s in MAC 20-5, 20-6, 20-7 and 20-8 are multiplication-accumulation units that perform convolution processing on the target region AR2 of the object data to which filters are applied.

[0159] Here, it is assumed that all pixel data in the target area AR2 are zero values. That is, pixel data b11, b12, b21, and b22 are all zero values.

[0160] In this case, the four MAC 20s in MAC 20-5, MAC 20-6, MAC 20-7 and MAC 20-8 do not need to perform multiplication and accumulation operations.

[0161] Therefore, the avoidance processing unit 21 avoids the convolution processing (multiplication and accumulation operation processing) of the target region AR2, and instead performs the convolution processing of the target region AR3.

[0162] That is, the pixel data c11, c12, c21, and c22 of the target area AR3 are input into four MAC 20s in MAC 20-5, MAC 20-6, MAC 20-7, and MAC 20-8 (see...). Figure 9 ).

[0163] In this way, when all pixel data in the target area AR is zero, the product operation processing of the target area AR is stopped, and MAC 20 is used for the product operation processing of another target area AR.

[0164] In addition, Figure 8 and Figure 9 For simplicity, the target regions AR1, AR2, and AR3 are shown as non-overlapping. However, depending on the filter's stride (offset), target regions AR1, AR2, and AR3 may sometimes partially overlap. For example, with a stride of "1", pixel data a12 of target region AR1 and pixel data b11 of target region AR2 are the same pixel data.

[0165] Description returned Figure 7 .

[0166] The multiplication-accumulation operation control unit 25 performs the processing of storing the operation result output from the MAC array unit 17 in the third memory 24. At this time, unless the relationship between the operation result output from the MAC array unit 17 and the target region AR is correctly correlated, the result of the convolution processing cannot be properly processed.

[0167] Therefore, when performing the process of avoiding unnecessary operations as described above, the avoidance processing unit 21 notifies the multiplication and accumulation operation control unit 25 of information for specifying the avoidance operation or information for specifying which target region AR the operation performed using the MAC array unit 17 belongs to.

[0168] Upon receiving the notification, the multiplication-accumulation control unit 25 stores the result of the multiplication-accumulation operation in the third memory 24. At this time, for the product operation results that have been avoided, a zero value is stored in the third memory 24.

[0169] Therefore, the multiplication operation control unit 25 can appropriately process the operation results output from the MAC array unit 17.

[0170] Note that, as a method to skip operations when both the input data and the result are set to zero, there are methods that store only non-zero values ​​with assigned addresses in memory and do not store zero values ​​in memory (see, for example, Patent Document 1). In this case, when the quantization bit length of the input data is large, memory utilization efficiency can be improved and power consumption reduced by assigning addresses and selectively storing the input data in memory.

[0171] However, reducing the quantization bit length of the input data has been considered to improve computational speed (image recognition processing speed) and power consumption. When the quantization bit length is reduced, the quantization bit length of the input data is ultimately set to 1 bit.

[0172] In this case, in the method of storing only non-zero values ​​in memory in association with addresses, if the input data of non-zero values ​​is not very large, the effect of improving memory utilization efficiency is reduced or not achieved at all.

[0173] Specifically, given a quantization bit length of N (bits), an address bit rate of Log(2, number of data rows), and a non-zero rate of R, the necessary storage amount is represented by the following expression (2). Here, "2" (2, number of data rows) in Log represents the base, and "number of data rows" represents the argument.

[0174] Number of data rows × N × Log(2, number of data rows) × R...... Expression (2)

[0175] As understood from expression (2), in the method of storing only non-zero values ​​in memory in association with addresses, memory utilization efficiency cannot be improved in the case of N=1 unless the value of R is small.

[0176] According to this configuration, since no address is added, even with a reduction in the quantization bit length of the input data, it is possible to reliably achieve the effect of improving memory utilization efficiency and reducing power consumption by the amount obtained by skipping multiplication and accumulation operations.

[0177] <2-2. Configuration Example 2>

[0178] Figure 10 A specific configuration of the signal processing unit 14B in configuration example 2 is shown.

[0179] The signal processing unit 14B in Configuration Example 2 is configured to avoid multiplication and accumulation operations associated with the weight data w when a portion of the weight data w in the filter F is zero. That is, the signal processing unit 14B includes a second avoidance processing unit 21b.

[0180] Figure 11 The filter F2 in this embodiment is shown. Figure 12 The diagram shows the processing of object data and object regions AR4, AR5, and AR6.

[0181] Filter F2 has three pixels vertically and horizontally. Therefore, the target regions AR4, AR5, and AR6 are also three-pixel regions in both the vertical and horizontal directions.

[0182] The values ​​of weight data w11, w12, w13, w22, w31, w32, and w33 in filter F2 are "1", and the values ​​of weight data w21 and w23 are "0".

[0183] Target area AR4 is defined as pixel data d11, d12, d13, d21, d22, d23, d31, d32, d33. Target area AR5 is defined as pixel data e11, e12, e13, e21, e22, e23, e31, e32, e33. Target area AR6 includes pixel data f11, f12, f13, f21, f22, f23, f31, f32, and f33.

[0184] The processing target data stored in the first memory 22 is input to each MAC 20 of the MAC array unit 17 via the first avoidance processing unit 21a (see Figure 10 ).

[0185] The weight data stored in the second memory 23 is input to each MAC 20 of the MAC array unit 17 via the second avoidance processing unit 21b.

[0186] Input the weight data w11 (=1) into MAC 20-1, the weight data w12 (=1) into MAC 20-2, the weight data w13 (=1) into MAC 20-3, and the weight data w21 (=0) into MAC 20-4 (see...). Figure 13 ).

[0187] Here, the multiplication process associated with the weight data w21 becomes zero, regardless of the pixel data, and is therefore avoided.

[0188] Therefore, the second avoidance processing unit 21b stops using the multiplication and accumulation operation of the weight data w21, and instead uses the weight data w22 to perform the multiplication and accumulation operation (see...). Figure 14 ).

[0189] In addition, the second avoidance processing unit 21b notifies the first avoidance processing unit 21a of the weight data w21 that has been avoided and the weight data w22 that has been newly adopted (see [link]). Figure 10 ).

[0190] The first avoidance processing unit 21a stops the input of pixel data d21, e21, and f21, which will be scheduled to be used in the multiplication process associated with the weight data w22, into MAC 20-4, MAC 20-8, and MAC 20-12, and determines that pixel data d22, e22, and f22, which will be used in the multiplication process associated with the alternative weight data w22, will be input into MAC 20-4, MAC 20-8, and MAC 20-12 (see...). Figure 14 ).

[0191] That is, the pixel data and weight data w input to the MAC array unit 17 are as follows: Figure 15 As shown.

[0192] Furthermore, the first avoidance processing unit 21a notifies the multiplication-accumulation control unit 25 of the pixel data d21, e21, f21 that avoids multiplication-accumulation operations and the pixel data d22, e22, f22 used for multiplication-accumulation operations, thus enabling the multiplication-accumulation control unit 25 to appropriately process the calculation result. Alternatively, the first avoidance processing unit 21a may not notify the pixel data but instead notify the multiplication-accumulation control unit 25 of the weight data w that avoids multiplication-accumulation operations and the weight data w used.

[0193] The multiplication and accumulation control unit 25 stores the multiplication and accumulation result output from the MAC array unit 17 into the third memory 24. At this time, for the product result that has been avoided, a zero value is stored in the third memory 24.

[0194] Therefore, the multiplication operation control unit 25 can appropriately process the operation results output from the MAC array unit 17.

[0195] Note that in Figure 13 , Figure 14 and Figure 15 In this example, weight data set to zero and its corresponding pixel data are shown as being temporarily loaded into the first local memory 26 and the second local memory 27. However, in practice, it is also possible to determine whether the pixel data is zero or whether the pixel data is the corresponding pixel data before loading the pixel data into the first local memory 26 or the second local memory 27. In this case, the weight data with zero values ​​and the corresponding pixel data are not loaded into the first local memory 26 or the second local memory 27.

[0196] <2-3. Configuration Example 3>

[0197] The signal processing unit 14C in Configuration Example 3 has a configuration for applying multiple filters F3, F4 and F5 to a target area AR.

[0198] Specifically, will refer to Figure 16 As an example, four target regions AR7, AR8, AR9, and AR10, and three filters F3, F4, and F5 are described.

[0199] Target regions AR7, AR8, AR9, and AR10 are regions consisting of two pixels each in the top, bottom, left, and right directions. Target region AR7 includes pixel data g11, g12, g21, and g22. Similarly, target region AR8 includes pixel data h11, h12, h21, and h22, target region AR9 includes pixel data i11, i12, i21, and i22, and target region AR10 includes pixel data j11, j12, j21, and j22.

[0200] The filters F3, F4, and F5 applied to the target areas AR7, AR8, AR9, and AR10 also have a size of 2 pixels in both the vertical and horizontal directions.

[0201] Filter F3 includes weight data wa11, wa12, wa21 and wa22, filter F4 includes weight data wb11, wb12, wb21 and wb22, and filter F5 includes weight data wc11, wc12, wc21 and wc22.

[0202] For example, by applying filter F3 to the target region AR7, the operation g11×wa11+g12×wa12+g21×wa21+g22×wa22 is performed. Furthermore, by applying filter F4 to the target region AR7, the operation g11×wb11+g12×wb12+g21×wb21+g22×wb22 is performed. Then, by applying filter F5 to the target region AR7, the operation g11×wc11+g12×wc12+g21×wc21+g22×wc22 is performed.

[0203] Then, in the convolution operation, a result is obtained by adding the results obtained by applying filter F3 to the target region AR7, the results obtained by applying filter F4 to the target region AR7, and the results obtained by applying filter F5 to the target region AR7.

[0204] Figure 17 An example configuration of the signal processing unit 14C is shown when performing this convolution process.

[0205] The signal processing unit 14C includes a first memory 22 and an avoidance processing unit 21, and the avoidance processing unit 21 performs the process of loading pixel data stored in the first memory 22 into the first local memory 26.

[0206] Therefore, the pixel data g11 of target region AR7, the pixel data h11 of target region AR8, the pixel data i11 of target region AR9, and the pixel data j11 of target region AR10 are loaded into the first local memory 26.

[0207] The signal processing unit 14C includes a second memory 23 and a second local memory 27, and loads weight data stored in the second memory 23 into the second local memory 27.

[0208] Therefore, the weight data wa11 of filter F3, the weight data wb11 of filter F4, and the weight data wc11 of filter F5 are loaded into the second local memory 27.

[0209] Meanwhile, in the convolution processing of the target region AR7, four multiplication operations are required for each filter F, that is, a total of 12 operations. For example... Figure 17 As shown, when performing one operation using MAC array unit 17, three multiplication operations are performed out of 12 operations.

[0210] Therefore, in order to complete the convolution processing of the target region AR7, four operations using the MAC array unit 17 are required.

[0211] For example, Figure 18 The second computational processing for the target region AR7 using MAC array unit 17 is shown.

[0212] like Figure 17 and Figure 18 As shown, the convolution process in this example can be achieved by repeatedly performing multiplication and accumulation operations using MAC array cell 17.

[0213] Here, note the pixel data entered for each MAC 20. Figure 17 The pixel data points g11, h11, i11, and j11 shown are all "1". On the other hand, in Figure 18 The pixel data h12, i12 and j12 shown are "1", but the pixel data g12 is zero.

[0214] In this case, since the processing result becomes zero regardless of the weight data w, there is no need to perform the multiplication process in the three MAC 20 of the input pixel data g12.

[0215] Therefore, the processing unit 21 avoids loading pixel data of another target area AR into the first local memory 26, instead of loading pixel data g12 into the first local memory 26.

[0216] That is, to obtain such Figure 19 The state is shown. Additionally, pixel data k12 refers to pixel data outside the target area AR7, AR8, AR9, and AR10.

[0217] In this way, data is loaded into the first local memory 26 while avoiding setting pixel data to zero.

[0218] The avoidance processing unit 21 notifies the multiplication-accumulation operation control unit 25 of information specifying pixel data that has not been loaded into the first local memory 26. The multiplication-accumulation operation control unit 25 adds zero to the avoided multiplication-accumulation operation result and stores it in the third memory 24.

[0219] Therefore, the multiplication operation control unit 25 can appropriately process the operation results output from the MAC array unit 17.

[0220] In addition, Figure 17 , Figure 18 as well as Figure 19The example shown illustrates an avoidance processing unit 21 that determines whether pixel data is zero and selects the pixel data to be loaded into the first local memory 26. However, it is also possible to set up an avoidance processing unit 21 that determines whether weight data is zero and selects the weight data to be loaded into the second local memory 27. In this case, an avoidance processing unit 21 related to pixel data, an avoidance processing unit 21 related to weight data, or only an avoidance processing unit 21 related to weight data can be set.

[0221] <2-4. Configuration Example 4>

[0222] In the signal processing unit 14D in configuration example 4, an avoidance processing unit 21D is set for each MAC 20D.

[0223] Specifically, such as Figure 20 As shown, pixel data is loaded from the first memory 22 to the first local memory 26 without passing through the avoidance processing unit 21. Additionally, weight data is loaded from the second memory 23 to the second local memory 27 without passing through the avoidance processing unit 21.

[0224] Pixel data and weight data are input from the first local memory 26 and the second local memory 27 to the corresponding MAC 20D.

[0225] In addition to adder and multiplier circuits, the MAC 20D includes, for example, Figure 21 The avoidance processing unit 21D and the zero value output unit 28 are shown.

[0226] The avoidance processing unit 21D determines whether the input pixel data is zero. If it is determined that the pixel data is zero, the clock applied to MAC 20D is stopped, and the zero value output unit 28 performs calculations to output a zero value as output data.

[0227] The avoidance processing unit 21D and the zero-value output unit 28 can be constructed from logic circuits, etc. For example, the zero-value output unit 28 can force the output value to be set to zero using a zero-value AND circuit.

[0228] By stopping the clock when the input pixel data is zero, the power consumption of the MAC 20D can be suppressed and power saving can be achieved.

[0229] Note that instead of determining whether the input pixel data is zero, it is possible to determine whether the input weight data is zero. Then, if the weight data is zero, clock stopping and zero-value output processing can be performed.

[0230] Of course, both the input pixel data and the input weight data can be monitored, and clock stop and zero-value output processing can be performed if at least one of the pixel data or weight data is zero.

[0231] It should be noted that in the signal processing unit 14D in configuration example 4, in the next stage, the result of the avoided multiplication-accumulation operation is output as zero to the MAC 20D or the multiplication-accumulation operation control unit 25. Therefore, it is not necessary to notify the multiplication-accumulation operation control unit 25 of the information used to specify the avoided multiplication-accumulation operation.

[0232] <3. Flowchart>

[0233] The processing flow used to implement the above examples is shown as a flowchart.

[0234] <3-1. First Processing Example>

[0235] In the first processing example, it is determined whether the pixel data is zero to appropriately avoid multiplication accumulation operations. For example, configuration example 1 of signal processing unit 14A can be implemented by executing the first processing example.

[0236] exist Figure 22 In step S100, the signal processing unit 14A retrieves weight data from the second memory 23 and loads the weight data into the second local memory 27.

[0237] In step S101, the signal processing unit 14A retrieves pixel data from the first memory 22. Subsequently, in step S102, the signal processing unit 14A determines whether a predetermined pixel data group includes data with non-zero values.

[0238] The specified pixel data group is, for example, Figure 8 The pixel data of target area AR1 shown are a11, a12, a21, a22, and the pixel data of target area AR2 are b11, b12, b21, b22, etc.

[0239] If non-zero data is not included in the predetermined pixel data group, that is, if all pixel data in the predetermined pixel data group are zero, the signal processing unit 14A (avoidance processing unit 21) notifies the multiplication-accumulation operation control unit 25 of information for specifying the operation to be avoided in step S103. Specifically, the multiplication-accumulation operation control unit 25 is notified of the position information (e.g., x and y coordinates) in the longitudinal and lateral directions for specifying the position of the control target area.

[0240] After notifying the multiplication and accumulation operation control unit 25, the signal processing unit 14A (avoidance processing unit 21) returns to the processing in step S101 to obtain the next pixel data.

[0241] On the other hand, if it is determined in step S102 that the predetermined pixel data group includes data with non-zero values, in step S104, the signal processing unit 14A (avoidance processing unit 21) loads the acquired pixel data into the first local memory 26.

[0242] In step S105, the signal processing unit 14A (avoidance processing unit 21) determines whether the loading of pixel data is complete. If it is determined that the loading of pixel data is not complete, the signal processing unit 14A (avoidance processing unit 21) returns to the processing in step S101 and obtains the next pixel data.

[0243] On the other hand, if it is determined in step S105 that the pixel data loading has been completed, the signal processing unit 14A performs a multiplication-accumulation operation in step S106. This process is performed at regular intervals, preparing the data required for the multiplication-accumulation operation in each of the first local memory 26 and the second local memory 27.

[0244] In step S107, the signal processing unit 14A sends the calculation result to the multiplication and accumulation operation control unit 25.

[0245] In step S108, the signal processing unit 14A (multiplication and accumulation operation control unit 25) compensates for the zero value of the operation result as an avoidance operation. This prevents the loss of the operation result of the avoided operation.

[0246] In step S109, the signal processing unit 14A (multiplication and accumulation operation control unit 25) performs the process of storing the operation result in the third memory 24.

[0247] In step S110, the signal processing unit 14A (multiplication and accumulation operation control unit 25) determines whether all operations have been completed. If the operations have not been completed, in step S109, for the new image data and the data stored in the third memory 24 as the result of the operations, the series of processes that started from step S100 are executed again.

[0248] On the other hand, when it is determined in step S110 that all operations have been completed, the signal processing unit 14A (multiplication and accumulation operation control unit 25) ends. Figure 22 The series of processes shown in the figure. At this time, the process of outputting the final calculation result stored in the third memory 24 to the outside of the signal processing unit 14A can be performed.

[0249] <3-2. Second Processing Example>

[0250] In the second processing example, it is determined whether the pixel data is zero to appropriately avoid multiplication accumulation operations, and it is also determined whether the weight data is zero to appropriately avoid multiplication accumulation operations. For example, configuration example 2 of signal processing unit 14B can be implemented by executing the second processing example.

[0251] It should be noted that processes similar to those in the first processing example are represented by the same step numbers, and their descriptions will be omitted appropriately.

[0252] exist Figure 23 In step S201, the signal processing unit 14B (second avoidance processing unit 21b) retrieves weight data from the second memory 23.

[0253] In step S202, the signal processing unit 14B (second avoidance processing unit 21b) determines whether the acquired weight data is zero. If it is determined to be zero, in step S203, the signal processing unit 14B (second avoidance processing unit 21b) notifies the multiplication and accumulation operation control unit 25 of the position information of the weight data.

[0254] After notifying the multiplication and accumulation control unit 25, the signal processing unit 14B (second avoidance processing unit 21b) returns to the processing in step S201 and acquires the next pixel data.

[0255] On the other hand, when it is determined that the acquired weight data is not zero, in step S204, the signal processing unit 14B (second avoidance processing unit 21b) loads the acquired weight data into the second local memory 27.

[0256] In step S205, the signal processing unit 14B (second avoidance processing unit 21b) determines whether the loading of weight data has been completed. If it is determined that the loading of weight data has not been completed, the signal processing unit 14B (second avoidance processing unit 21b) returns to the processing in step S201 and obtains the next weight data.

[0257] On the other hand, if it is determined in step S205 that the loading of weight data has been completed, in step S101, the signal processing unit 14B (first avoidance processing unit 21a) obtains pixel data from the first memory 22.

[0258] In step S206, the signal processing unit 14B (first avoidance processing unit 21a) determines whether the acquired pixel data corresponds to the weight data that was determined to be zero, i.e., the weight data that was not loaded into the second local memory 27. For example, the corresponding pixel data is Figure 13 The pixel data shown is d21, e21, f21, etc.

[0259] If it is determined that the acquired pixel data corresponds to the weight data that is determined to be zero, in step S101, the signal processing unit 14B (first avoidance processing unit 21a) acquires new pixel data without loading the acquired pixel data into the first local memory 26.

[0260] On the other hand, if it is determined that the acquired pixel data does not correspond to the weight data that is determined to be zero, in step S207, the signal processing unit 14B (first avoidance processing unit 21a) determines whether the acquired pixel data is zero. If it is determined that the acquired pixel data is zero, in step S208, the signal processing unit 14B (first avoidance processing unit 21a) notifies the multiplication and accumulation operation control unit 25 of the pixel data's position information. That is, the acquired pixel data is not loaded into the first local memory 26.

[0261] If the acquired pixel data does not correspond to the weight data that is set to zero and is not zero, in step S104, the signal processing unit 14B (first avoidance processing unit 21a) loads the acquired pixel data into the first local memory 26.

[0262] Subsequently, in step S105, the signal processing unit 14B (first avoidance processing unit 21a) determines whether the loading of pixel data has been completed. If it is determined that the loading of pixel data has not been completed, the signal processing unit 14B (first avoidance processing unit 21a) returns to the processing in step S101 and obtains the next pixel data.

[0263] On the other hand, if it is determined in step S105 that the pixel data loading has been completed, the signal processing unit 14B then... Figure 24 In step S106, a multiplication and accumulation operation is performed, and in step S107, the operation result is transmitted to the multiplication and accumulation operation control unit 25.

[0264] Subsequently, the signal processing unit 14B (multiplication and accumulation operation control unit 25) compensates for the zero value of the operation result as an avoidance operation in step S108, and performs the process of storing the operation result in the third memory 24 in step S109.

[0265] In step S110, the signal processing unit 14B (multiplication and accumulation operation control unit 25) determines whether all operations have been completed. If the operations have not been completed, the processing returns to the processing in step S201 to perform a new multiplication and accumulation operation.

[0266] On the other hand, if it is determined in step S110 that all operations have been completed, the signal processing unit 14B (multiplication and accumulation operation control unit 25) ends. Figure 23 and Figure 24 The series of processes shown in the figure. At this time, the final calculation result stored in the third memory 24 can be output to the outside of the signal processing unit 14B.

[0267] Note that in convolutional processing and other applications with a large number of target regions (ARs), there may be cases where the multiplication and accumulation operation in step S106 is performed only once without ending the operation using the same filter F. In such cases, after the processing in step S110 is completed, the processing returns to... Figure 23 The process proceeds to step S101 without returning to step S201. Thus, the multiplication and accumulation operation is performed appropriately.

[0268] <3-3. Third Processing Example>

[0269] The third processing example is a flowchart illustrating configuration example 4 for implementing signal processing unit 14D. Specifically, the third processing example is a configuration for setting up avoidance processing unit 21D and zero-value output unit 28 for each MAC 20D.

[0270] It should be noted that processes similar to those in the first processing example are represented by the same step numbers, and their descriptions will be omitted appropriately.

[0271] Signal processing unit 14D (multiplication and accumulation operation control unit 25) in Figure 25 In step S100, weight data is obtained from the second memory 23 and loaded into the second local memory 27.

[0272] Next, in step S301, the signal processing unit 14D (multiplication and accumulation operation control unit 25) retrieves pixel data from the first memory 22 and loads the pixel data into the first local memory 26.

[0273] In step S302, the signal processing unit 14D (avoidance processing unit 21D) determines whether the input pixel data is a zero value. This processing is performed for each MAC 20D.

[0274] In the MAC 20D that determines the input pixel data to be zero, in step S303, the signal processing unit 14D (avoidance processing unit 21D) performs clock stop processing. Furthermore, in step S304, the signal processing unit 14D (avoidance processing unit 21D) causes the zero-value output unit 28 to perform zero-value output processing. Therefore, in the MAC 20D, multiplication and accumulation operations are avoided, and power consumption is reduced.

[0275] In addition, the MAC 20D outputs a zero value as the calculation result.

[0276] On the other hand, in the MAC 20D that determines that the input pixel data is not zero, the signal processing unit 14D performs a multiplication and accumulation operation in step S106.

[0277] Therefore, a multiplication and accumulation operation is performed on the pixel data and weight data that serve as input data.

[0278] After completing step S304 or after completing step S106, in step S107, the signal processing unit 14D sends the calculation result to the multiplication and accumulation operation control unit 25.

[0279] In step S109, the signal processing unit 14D (multiplication and accumulation operation control unit 25) performs the process of storing the operation result in the third memory 24.

[0280] In step S110, the signal processing unit 14D (multiplication-accumulation operation control unit 25) determines whether all operations have been completed. If the operations are not completed, in step S109, for the new image data and the data as the operation result stored in the third memory 24, the operation is executed again from... Figure 25 A series of processes starting from step S100.

[0281] On the other hand, if it is determined in step S110 that all operations have been completed, the signal processing unit 14D (multiplication-accumulation operation control unit 25) ends. Figure 22 The series of processes shown in the figure.

[0282] <4. Variations>

[0283] Modifications to the above embodiments will be described.

[0284] <4-1. First Variation>

[0285] In each example, when the input data for pixel data and weight data are zero, a process to avoid multiplication and accumulation operations related to the data has been described.

[0286] For example, when the image data includes as many zero values ​​as the edge image, the number of operations can be effectively reduced, and the power consumption of the MAC array unit 17 can be reduced.

[0287] However, image data does not necessarily contain many zero values. In this case, if the configuration is set to avoid multiplication and accumulation operations when the input data has zero values, the number of multiplication and accumulation operations that can be avoided is smaller, and thus the power reduction effect is reduced.

[0288] Therefore, when the input data is less than a predetermined threshold, it is conceivable to treat the input data as zero, thereby reducing the number of avoidable product operations.

[0289] For example, when pixel data is represented by 4 bits, that is, when pixel data is any value from 0 to 15, and when the predetermined threshold is "4" and pixel data is from 0 to 3, multiplication operations related to pixel data are avoided. Of course, the predetermined threshold "4" is an example and can be any number, such as "8" or "10".

[0290] This means, for example, in edge images, ignoring weak edge pixels (pixels with small differences from neighboring pixels) and performing convolution processing based on strong edge pixels (pixels with large differences from neighboring pixels). Therefore, memory usage efficiency can be improved and power consumption can be reduced when performing image recognition processing based on stronger features.

[0291] It should be noted that, in implementing this variant, in Figure 22 In step S102, instead of determining whether the predetermined pixel data group includes non-zero values, it is only necessary to determine whether the predetermined pixel data group includes pixel data that is equal to or greater than a predetermined threshold.

[0292] Furthermore, as in configuration example 2 of signal processing unit 14B, when the weight data is less than a predetermined threshold, multiplication and accumulation operations can be avoided by treating not only the pixel data but also the weight data as zero. In this case, the predetermined threshold used to determine the pixel data and the predetermined threshold used to determine the weight data can be different. For example, the predetermined threshold used to determine the pixel data can be set to a first threshold (e.g., "4"), and the predetermined threshold used to determine the weight data can be set to a second threshold (e.g., "2").

[0293] In application Figure 23 In the case of the flowchart, instead of determining whether the weight data is zero in step S202, it is determined whether the weight data is less than a predetermined threshold.

[0294] Then, in Figure 23 In step S206, it is determined whether the data corresponds to weight data that is determined to be less than a predetermined threshold, and in step S207, it is determined whether the pixel data is less than the predetermined threshold.

[0295] <4-2. Second Variation>

[0296] As a second modification, MAC20E is capable of performing operations within recurrent neural networks (RNNs). Specifically, MAC20E may include long short-term memory (LSTM) (see...). Figure 26 ).

[0297] In this case, such as Figure 26 As shown, the processing of each of the above embodiments can be achieved by setting the feedback output of the LSTM to OFF or setting the feedback output to zero.

[0298] <4-3. Variations of the sensor unit>

[0299] for Figure 2 The construction of the sensor unit shown is subject to several modifications. For example, in each of the examples above, sensor unit 3, used as a DVS, has been described as an example, but sensor unit 3 could be a sensor unit that generates image data by reading grayscale signals from pixel 16 rather than detecting the presence or absence of an event. In this case, this is from Figure 2 Remove the configuration of arbitrator 12.

[0300] In addition, such as Figure 27 As shown, the signal processing unit 14F, including the avoidance processing unit 21, can be disposed outside the sensor unit 3.

[0301] Specifically, the sensor unit 3F includes a pixel array unit 11, a readout unit 13, a preprocessing unit 29, and an output unit 15, with the output unit 15 connected to the bus 30. The preprocessing unit 29 is the unit that performs signal processing as preprocessing in each of the various types of processing performed by the signal processing unit 14 in each of the above examples.

[0302] The control unit 4, which includes the memory 31 and the signal processing unit 14F, is connected to the bus 30. That is, the signal processing unit 14F, which includes the aforementioned avoidance processing unit 21, is provided outside the sensor unit 3F.

[0303] In addition, such as Figure 28 As shown, the signal processing unit 14F, including the avoidance processing unit 21, can be disposed outside the sensor unit 3F and outside the control unit 4.

[0304] Specifically, the sensor unit 3F includes a pixel array unit 11, a readout unit 13, a preprocessing unit 29, and an output unit 15, with the output unit 15 connected to the bus 30.

[0305] The control unit 4, memory 31 and signal processing unit 14F are connected to bus 30.

[0306] The signal processing unit 14F includes a MAC array unit 17, a signal processing control unit 18 including an avoidance processing unit 21, a memory unit 19, etc.

[0307] In addition, such as Figure 29 As shown, the signal processing unit 14F, including the avoidance processing unit 21, can be disposed in another signal processing device.

[0308] Specifically, for example, the various functions described above can be achieved by an imaging device 1 including a sensor unit 3F, a control unit 4, a memory 31, and a communication unit 32, and by another signal processing device 34 including a signal processing unit 14F and a communication unit 32.

[0309] The communication unit 32 of the imaging device 1 can perform wired or wireless data communication with the communication unit 33 of another signal processing device 34.

[0310] By adopting such various configurations, the various functions of the aforementioned signal processing unit can be realized.

[0311] <4-4 Other modifications>

[0312] The examples above have described an instance of performing signal processing on two-dimensional data such as image data, but the target of the processing can be one-dimensional data.

[0313] One-dimensional data includes, for example, sound data, output data such as velocity data, acceleration data and angular velocity data from a gyroscope sensor, and position information.

[0314] These one-dimensional data can be arranged along different dimensional directions for each predetermined amount of data to form two-dimensional data.

[0315] These data segments can be transformed into data containing many zero values ​​by being converted relative to a reference value. By performing this transformation process, the aforementioned power savings can be achieved at a higher level.

[0316] <5. Summary>

[0317] As described above, the imaging apparatus 1, which is a signal processing device, includes a multiplication-accumulation operation unit (MAC 20, 20D, and 20E) arranged in a one-dimensional or two-dimensional array and capable of performing multiplication-accumulation operations in a neural network, a threshold determination processing unit (avoidance processing units 21 and 21D, a first avoidance processing unit 21a, and a second avoidance processing unit 21b) that determines whether the input data (pixel data and weight data) used for the operation by the multiplication-accumulation operation unit is less than a predetermined threshold, and avoidance processing units 21 and 21D (first avoidance processing unit 21a and second avoidance processing unit 21b) that avoid performing multiplication-accumulation operations on the input data when the input data is less than the predetermined threshold.

[0318] Input data that is less than a predetermined threshold is, for example, input data that is zero or close to zero. To determine whether the value is zero, the threshold is set to "1", and then it is determined whether the input data is less than the threshold.

[0319] When the input data is zero, the result of the multiplication and accumulation operation is obviously zero, and the calculation can be performed without performing the multiplication and accumulation operation. According to this configuration, since the multiplication and accumulation operation is avoided when the input data is zero, the multiplication and accumulation unit is prevented from performing useless calculations, and power consumption is reduced.

[0320] As described in the signal processing unit 14A, etc., in Configuration Example 1, the input data may include a first type of input data (pixel data) and a second type of input data (weight data). The threshold determination processing unit (avoidance processing units 21 and 21D, first avoidance processing unit 21a and second avoidance processing unit 21b) can perform determination for the first type of input data, and if the first type of input data is less than a predetermined threshold, the avoidance processing units 21 and 21D (first avoidance processing unit 21a and second avoidance processing unit 21b) can avoid multiplication and accumulation operation processing for the first type of input data.

[0321] It should be noted that in the description of configuration example 1, a predetermined threshold is set to "1" to determine whether the first type of input data is a zero value.

[0322] The multiplication-accumulation unit (MAC 20, 20D, and 20E) multiplies the first type of input data with the second type of input data. That is, if either the first type of input data or the second type of input data is zero, the result of the multiplication is also zero. This configuration avoids multiplication-accumulation operations when the first type of input data is zero.

[0323] According to this configuration, since multiplication and accumulation operations are avoided when the first type of input data is zero, multiplication and accumulation operations with zero results can be effectively avoided.

[0324] As described in each example of the signal processing unit 14A in configuration example 1, the second type of input data can be weight data, as information about the weights multiplied by the first type of input data (pixel data).

[0325] Weight data, for example, are the coefficients of filters applied to a predetermined range of image data in a CNN. It is difficult to consider filters where all filter coefficients are zero.

[0326] Therefore, for example, by performing a determination process regarding whether the first type of input data of the image data set to a predetermined area is zero and appropriately avoiding multiplication and accumulation operations, unnecessary multiplication and accumulation operations can be effectively eliminated and power saving can be achieved.

[0327] As described in Configuration Example 1, a threshold determination processing unit (avoidance processing unit 21, 21D, first avoidance processing unit 21a, and second avoidance processing unit 21b) can be set for each of the multiple multiplication accumulation operation units (MAC 20, 20D, and 20E).

[0328] Determine whether each of the multiple input data input to multiple multiplication and accumulation units is less than a predetermined threshold, for example, whether the input data is zero.

[0329] Therefore, processing such as replacing input data determined to be less than a predetermined threshold can be performed, and the multiplication and accumulation unit can be used efficiently. In other words, the expansion and usage of the multiplication and accumulation unit until a predetermined result is obtained can be reduced, which can help reduce power consumption.

[0330] As described in Configuration Example 1, Configuration Example 2, Configuration Example 3, etc., the avoidance processing units 21 and 21D (first avoidance processing unit 21a and second avoidance processing unit 21b) can change the input data to the multiplication and accumulation operation unit (MAC 20, 20D and 20E) in a way that avoids performing multiplication and accumulation operations on input data less than a predetermined threshold when the input data (pixel data and weight data) is less than a predetermined threshold.

[0331] Therefore, input data that is equal to or greater than a predetermined threshold is input to the multiplication and accumulation unit.

[0332] Therefore, the multiplication and accumulation unit can be used effectively, and unnecessary multiplication and accumulation operations can be prevented.

[0333] As described in Configuration Example 1, Configuration Example 2, Configuration Example 3, etc., a multi-cumulative operation control unit 25 can be provided to manage the input data (pixel data and weight data) and output data of the multi-cumulative operation processing, and avoidance processing units 21 and 21D (first avoidance processing unit 21a and second avoidance processing unit 21b) can notify the multi-cumulative operation control unit 25 of information for specifying the input data that has been avoided from the multi-cumulative operation processing.

[0334] Therefore, the multiplication and accumulation operation control unit 25 can grasp the correspondence between the input data used in the multiplication and accumulation operation and the result of the multiplication and accumulation operation.

[0335] Therefore, the computation results can be processed appropriately, and for example, convolution processing in CNNs can be performed correctly. Furthermore, power saving is achieved by avoiding unnecessary multiplication and accumulation operations that result in zero values.

[0336] As described in Configuration Example 4, avoidance processing units 21 and 21D (first avoidance processing unit 21a and second avoidance processing unit 21b) can be set for each multiplication accumulation operation unit (MAC 20, 20D and 20E).

[0337] By providing an avoidance processing unit 21 for each multiplication and accumulation operation unit, the processing load of the determination process performed by one avoidance processing unit 21 can be reduced. In this determination process, it is determined whether the input data (pixel data and weight data) is less than a predetermined threshold, for example, whether the input data is zero.

[0338] Thus, for example, multiplication and accumulation operations can be avoided without performing processes such as replacing input data with non-zero values. Therefore, power saving can be achieved through simple processing.

[0339] As described in Configuration Example 4, the avoidance processing unit 21D can avoid performing multiplication and accumulation operations on input data (pixel data and weight data) that are less than a predetermined threshold, and output zero value as the processing result of the multiplication and accumulation operation.

[0340] For example, it is obvious that the result of the operation will be zero when the input data is zero, so we should avoid forcing the output data to zero after the multiplication and accumulation operation.

[0341] Therefore, the correct output data can be obtained as the result of the multiplication and accumulation operation, and power consumption can be reduced by avoiding the operation process.

[0342] The input data includes a first type of input data (pixel data) and a second type of input data (weight data). If the first type of input data is less than a first threshold, the avoidance processing units 21 and 21D (first avoidance processing unit 21a and second avoidance processing unit 21b) can change the first type of input data input to the multiplication and accumulation operation units (MAC 20, 20D and 20E) and notify the multiplication and accumulation operation control unit 25 of the information for specifying the changed first type of input data.

[0343] Therefore, the comparison process with a predetermined threshold can be performed only for one of the first type of input data and the second type of input data.

[0344] Therefore, compared to performing deterministic processing on both the first type of input data and the second type of input data, the processing load can be reduced and power consumption can be reduced.

[0345] If the first variation is applied to configuration example 2, when the second type of input data (weight data) is less than the second threshold, the avoidance processing unit (first avoidance processing unit 21a and second avoidance processing unit 21b) can change the second type of input data input to the multiplication-accumulation operation unit (MAC 20) and change the first type of input data (pixel data) corresponding to the changed second type of input data, and notify the multiplication-accumulation operation control unit 25 of the information for specifying the changed first type of input data and the changed second type of input data.

[0346] In multiplication and accumulation operations, the corresponding data is the number to be multiplied. During multiplication, if any of the numbers to be multiplied is zero, the result is zero, regardless of the value of the numbers being multiplied. To omit this multiplication process, the number of multiplications that result in zero values ​​(second-type input data) and the corresponding number of multiplications are omitted.

[0347] Therefore, when the second type of input data is zero, multiplication and subsequent addition are avoided, and multiplication and addition operations that result in non-zero values ​​can be performed in advance. Furthermore, since the multiplication-accumulation control unit can identify and avoid multiplication and addition operations, the result of the multiplication-accumulation operation can be processed appropriately. Additionally, reducing the number of multiplication and addition operations performed to obtain a specific result helps save power.

[0348] As described in Configuration Example 2, the multiplication-accumulation operation control unit 25 can manage the multiplication-accumulation operation results of the first type of input data (pixel data) and the second type of input data (weight data), and compensate for zero values ​​for the avoided multiplication-accumulation operation results.

[0349] The multiply-accumulate operation to be avoided (i.e., the multiply-accumulate operation to be skipped) can be specified by receiving information for specifying the corresponding first type of input data and second type of input data.

[0350] Then, the result of the specified multiplication-accumulation operation can be obtained by supplementing and managing zero values, ensuring that there is no lack of data. Therefore, convolution operations and other operations in CNNs can be performed efficiently while saving power.

[0351] For reference Figure 1 , Figure 2 , Figure 3As described above, the imaging device 1 includes: a pixel array unit 11, wherein photoelectric conversion elements (pixels 16) are arranged in a one-dimensional or two-dimensional array; and a signal processing unit 14 (14A, 14B, 14C, 14D, and 14F), wherein input data (pixel data and weight data) based on the output signal of the pixel array unit 11 is input to the signal processing unit 14 (14A, 14B, 14C, 14D, and 14F), wherein the signal processing unit 14 includes a multiplication-accumulation operation unit (MAC) arranged in a one-dimensional or two-dimensional array and capable of performing multiplication-accumulation operations in a neural network. 20, 20D, and 20E), threshold determination processing units (avoidance processing units 21 and 21D, first avoidance processing unit 21a and second avoidance processing unit 21b), which determine whether the input data used for the operation by the multiplication and accumulation operation unit is less than a predetermined threshold, and avoidance processing units 21 and 21D (first avoidance processing unit 21a and second avoidance processing unit 21b), avoid performing multiplication and accumulation operation processing on the input data when the input data is less than the predetermined threshold.

[0352] Due to issues such as battery problems, the signal processing unit 14 included in the imaging device 1 needs to be power-efficient.

[0353] According to this configuration, it is preferred in imaging devices capable of performing at least a portion of convolution operations in CNNs, etc., because it can reduce the power consumed in multiplication-accumulation operation processing.

[0354] For reference Figure 1 , Figure 2 , Figure 3 As described above, the pixel array unit 11 and the signal processing unit 14 can be integrally formed.

[0355] Because the pixel array unit 11 and the signal processing unit 14 are integrated, the imaging device 1 can be miniaturized.

[0356] Therefore, the ease of processing of the imaging device 1 can be improved.

[0357] For reference Figure 2 As described above, the feature quantities extracted from the output signal of the pixel array unit 11 can be input to the signal processing unit 14 (14A, 14B, 14C, 14D and 14F) as input data.

[0358] Feature data typically includes data with zero values ​​or values ​​less than a predetermined threshold.

[0359] Therefore, in many cases, multiplication and accumulation operations can be performed efficiently, and the power consumption reduction effect can be further enhanced.

[0360] It should be noted that the effects described in this specification are merely illustrative and not limited, and other effects may be provided.

[0361] <6. This technology> (1)

[0363] A signal processing apparatus, comprising:

[0364] The multiplication and accumulation unit is arranged in a one-dimensional or two-dimensional array and is capable of performing multiplication and accumulation operations in a neural network.

[0365] A threshold determination processing unit determines whether the input data for the operation performed by the multiplication-accumulation unit is less than a predetermined threshold; and

[0366] The avoidance processing unit avoids multiplication and accumulation operations on the input data when the input data is less than the predetermined threshold. (2)

[0368] According to the signal processing apparatus described in (1) above, wherein,

[0369] The input data includes a first type of input data and a second type of input data.

[0370] The threshold determination processing unit performs a determination for the first type of input data, and

[0371] If the first type of input data is less than the predetermined threshold, the avoidance processing unit avoids the multiplication and accumulation operation processing for the first type of input data. (3)

[0373] According to the signal processing device described in (2) above, wherein,

[0374] The second type of input data is weighted data, which serves as the weight information multiplied by the first type of input data. (4)

[0376] The signal processing apparatus according to any one of (1) to (3) above, wherein,

[0377] A threshold determination processing unit is set for each of the plurality of multiplication and accumulation operation units. (5)

[0379] According to the signal processing device described in (4) above, wherein,

[0380] If the input data is less than the predetermined threshold, the avoidance processing unit changes the input data input to the multiplication and accumulation operation unit, thereby avoiding multiplication and accumulation operation processing for the input data that is less than the predetermined threshold. (6)

[0382] The signal processing apparatus according to (5) above further includes:

[0383] The multiplication-accumulation operation control unit manages the input and output data of the multiplication-accumulation operation processing, wherein...

[0384] The avoidance processing unit notifies the multiplication-accumulation operation control unit of information indicating that the input data has been avoided from the multiplication-accumulation operation. (7)

[0386] The signal processing apparatus according to any one of (1) to (6) above, wherein,

[0387] The avoidance processing unit is set for each of the multiplication accumulation operation units. (8)

[0389] According to the signal processing apparatus described above (7), wherein,

[0390] The avoidance processing unit avoids the multiplication-accumulation operation processing for input data that is less than the predetermined threshold, and the avoidance processing unit outputs a zero value as the processing result of the multiplication-accumulation operation processing. (9)

[0392] According to the signal processing apparatus described above (6), wherein,

[0393] The input data includes a first type of input data and a second type of input data, and

[0394] When the first type of input data is less than the first threshold, the avoidance processing unit

[0395] The first type of input data input to the multiplication and accumulation unit is changed, and information specifying the changed first type of input data is notified to the multiplication and accumulation control unit. (10)

[0397] According to the signal processing apparatus described above (9), wherein,

[0398] If the second type of input data is less than the second threshold, the avoidance processing unit changes the second type of input data input to the multiplication and accumulation unit and changes the first type of input data corresponding to the changed second type of input data. The avoidance processing unit then notifies the multiplication and accumulation control unit of the information used to specify the changed first type of input data and the changed second type of input data. (11)

[0400] According to the signal processing apparatus described above (10), wherein,

[0401] The multiplication and accumulation operation control unit manages the multiplication and accumulation operation results of the first type of input data and the second type of input data, and compensates for zero values ​​for avoided multiplication and accumulation operation results. (12)

[0403] An imaging device, comprising:

[0404] A pixel array unit in which photoelectric conversion elements are arranged in a one-dimensional or two-dimensional array;

[0405] The signal processing unit receives input data based on the output signal of the pixel array unit.

[0406] The signal processing unit includes:

[0407] The multiplication and accumulation unit is arranged in a one-dimensional or two-dimensional array and is capable of performing multiplication and accumulation operations in a neural network.

[0408] A threshold determination processing unit determines whether the input data for the operation performed by the multiplication-accumulation unit is less than a predetermined threshold; and

[0409] The avoidance processing unit avoids multiplication and accumulation operations on the input data when the input data is less than the predetermined threshold. (13)

[0411] According to the imaging device described above (12), wherein,

[0412] The pixel array unit and the signal processing unit are formed as a single unit. (14)

[0414] According to the imaging device described above (13), wherein,

[0415] The feature data extracted based on the output signal of the pixel array unit is input to the signal processing unit as the input data. (15)

[0417] A signal processing method executed by a signal processing device, the method comprising:

[0418] Determine whether the input data used for the multiplication-accumulation operation in the neural network is less than a predetermined threshold; and

[0419] If the input data is less than the predetermined threshold, multiplication and accumulation operations on the input data are avoided.

[0420] Reference Symbol List

[0421] 1. Imaging device (signal processing device)

[0422] 20, 20D, 20E MAC (Multiplication and Accumulation Unit)

[0423] 20-1, 20-2, 20-3, 20-4 MAC (Multiplication and Accumulation Unit)

[0424] 20-5, 20-6, 20-7, 20-8 MAC (Multiplication and Accumulation Unit)

[0425] 20-9, 20-10, 20-11, 20-12 MAC (Multiplication and Accumulation Unit)

[0426] 21, 21D Avoidance Processing Unit (Threshold Determination Processing Unit)

[0427] 21a First Avoidance Processing Unit (Threshold Determination Processing Unit)

[0428] 21b Second Avoidance Processing Unit (Threshold Determination Processing Unit)

[0429] 25 Multiplication and Accumulation Operation Control Unit

Claims

1. A signal processing apparatus, comprising: The multiplication and accumulation unit is arranged in a one-dimensional or two-dimensional array and is capable of performing multiplication and accumulation operations in a neural network. The threshold determination processing unit determines whether the input data used for the operation performed by the multiplication and accumulation operation unit is less than a predetermined threshold. The avoidance processing unit avoids multiplication and accumulation operations on the input data when the input data is less than the predetermined threshold. as well as The multiplication-accumulation operation control unit manages the input and output data of the multiplication-accumulation operation processing, wherein... The input data includes a first type of input data and a second type of input data. The first type of input data is pixel data, and the second type of input data is weight data, which serves as weight information for multiplying with the first type of input data. When the first type of input data is less than the first threshold, the avoidance processing unit To avoid multiplication and accumulation operations on the first type of input data, the next set of first type of input data is input to the multiplication and accumulation unit, and information specifying the next set of first type of input data is notified to the multiplication and accumulation control unit.

2. The signal processing apparatus according to claim 1, wherein, The threshold determination processing unit performs a determination for the first type of input data, and If the first type of input data is less than the predetermined threshold, the avoidance processing unit avoids the multiplication and accumulation operation processing for the first type of input data.

3. The signal processing apparatus according to claim 1, wherein, A threshold determination processing unit is set for each of the plurality of multiplication and accumulation operation units.

4. The signal processing apparatus according to claim 3, wherein, If the input data is less than the predetermined threshold, the avoidance processing unit changes the input data input to the multiplication and accumulation operation unit, thereby avoiding multiplication and accumulation operation processing for the input data that is less than the predetermined threshold.

5. The signal processing apparatus according to claim 4, wherein, The avoidance processing unit notifies the multiplication-accumulation operation control unit of information indicating that the input data has been avoided from the multiplication-accumulation operation.

6. The signal processing apparatus according to claim 1, wherein, The avoidance processing unit is set for each of the multiplication accumulation operation units.

7. The signal processing apparatus according to claim 6, wherein, The avoidance processing unit avoids the multiplication-accumulation operation processing for input data that is less than the predetermined threshold, and the avoidance processing unit outputs a zero value as the processing result of the multiplication-accumulation operation processing.

8. The signal processing apparatus according to claim 1, wherein, When the second type of input data is less than the second threshold, the avoidance processing unit avoids the multiplication and accumulation operation processing for the second type of input data, inputs the next set of second type input data to the multiplication and accumulation operation unit, and inputs the first type input data corresponding to the next set of second type input data to the multiplication and accumulation operation unit. The avoidance processing unit notifies the multiplication and accumulation operation control unit of the information used to specify the next set of second type input data and the first type input data corresponding to the next set of second type input data.

9. The signal processing apparatus according to claim 8, wherein, The multiplication and accumulation operation control unit manages the multiplication and accumulation operation results of the first type of input data and the second type of input data, and compensates for zero values ​​for avoided multiplication and accumulation operation results.

10. An imaging device, comprising: A pixel array unit in which photoelectric conversion elements are arranged in a one-dimensional or two-dimensional array; The signal processing unit receives input data based on the output signal of the pixel array unit. The signal processing unit includes: The multiplication and accumulation unit is arranged in a one-dimensional or two-dimensional array and is capable of performing multiplication and accumulation operations in a neural network. The threshold determination processing unit determines whether the input data used for the operation performed by the multiplication and accumulation operation unit is less than a predetermined threshold. The avoidance processing unit, when the input data is less than the predetermined threshold, avoids multiplication and accumulation operations on the input data; and The multiplication-accumulation operation control unit manages the input and output data of the multiplication-accumulation operation processing, wherein... The input data includes a first type of input data and a second type of input data. The first type of input data is pixel data, and the second type of input data is weight data, which serves as weight information for multiplying with the first type of input data. When the first type of input data is less than the first threshold, the avoidance processing unit To avoid multiplication and accumulation operations on the first type of input data, the next set of first type of input data is input to the multiplication and accumulation unit, and information specifying the next set of first type of input data is notified to the multiplication and accumulation control unit.

11. The imaging apparatus according to claim 10, wherein, The pixel array unit and the signal processing unit are formed as a single unit.

12. The imaging apparatus according to claim 11, wherein, The feature data extracted based on the output signal of the pixel array unit is input to the signal processing unit as the input data.

13. A signal processing method executed by a signal processing apparatus, the method comprising: Determine whether the input data used for multiplication-accumulation operations in a neural network is less than a predetermined threshold; If the input data is less than the predetermined threshold, avoid multiplication and accumulation operations on the input data. as well as Manage the input and output data of the multiplication-accumulation operation, wherein, The input data includes a first type of input data and a second type of input data. The first type of input data is pixel data, and the second type of input data is weight data, which serves as weight information for multiplying with the first type of input data. When the first type of input data is less than the first threshold, the processing unit is avoided. To avoid multiplication and accumulation operations on the first type of input data, the next set of first type of input data is input to the multiplication and accumulation unit, and the information specifying the next set of first type of input data is notified to the multiplication and accumulation control unit.

Images