DATA PROCESSING APPARATUS, DATA PROCESSING METHOD, AND DATA PROCESSING PROGRAM

Abstract

The present invention can reduce power consumption by skipping calculations made possible by increasing the equivalence ratio of outputs.　In this data processing device, a decimal point position control unit determines, as a first control, a decimal point position in which the decimal point position for each layer is moved such that an upper limit counter value is within the range of a first threshold value and a lower limit counter value is within the range of a second threshold value. A detection calculation unit performs a calculation process on the data to be processed at the time of processing; a saturation rate control unit performs, as a second control repeated by the decimal point position control unit, control for increasing a lower limit saturation rate proportional to the magnitude of the counted lower limit counter value, and instructs the decimal point position control unit to move at least the decimal point position, with respect to the result of the first control; and as the second control, the decimal point position control unit makes a predetermined determination on the basis of the instruction from the saturation rate control unit and metadata, determines the moved decimal point position for each layer, and implements the calculation process.

Description

[Technical field]

[0001] The disclosed technology relates to a data processing device, a data processing method, and a data processing program. [Background technology]

[0002] Various data processing technologies using multi-layer neural networks and deep learning have been developed. The range of applications is expanding to various fields such as recognition and detection. For example, object detection is a technology that detects metadata consisting of the position (square frame surrounding the object) and attributes (type of object such as person, car, etc.) of an object contained in an input image, and the recognition accuracy of each object. In recent years, there is a technology for detecting metadata based on the results of deep learning for object detection. For example, Non-Patent Document 1 discloses YOLO (You Only Look Once). In addition, Non-Patent Document 2 discloses SSD (Single Shot Multibox Detector). These technologies are being considered for application to object detection systems that require real-time performance such as autonomous driving and surveillance cameras.

[0003] From the viewpoint of network traffic load balancing and security protection, object detection devices for autonomous driving and surveillance cameras are being considered for installation at the edge of the network, i.e., on the terminal side. In order to install object detection devices at the edge of the network, it is essential that the devices are compact and consume less power.

[0004] In order to achieve miniaturization and low power consumption, a configuration has been proposed in which object detection processing based on deep learning is implemented in hardware and the data bit width of each data handled by a computing unit is reduced (see Non-Patent Document 3). The data includes input, output (feature map), weight (kernel), and bias. In general, when inference processing of deep learning is performed by software, each data used in product-sum calculation is treated as 32-bit floating-point data. This is because the range of values ​​that each data can take is wide, and the range differs for each image or for each layer such as the convolution layer that constitutes the convolutional neural network. Non-Patent Document 3 reports that the effect of reducing the circuit size and power can be obtained by determining the data bit width in each layer of the convolutional neural network in advance using statistical information and reducing it to 8 to 16 bits.

[0005] In contrast to these approaches, a method has been disclosed in which the data width is uniformly set to a fixed-point number of n bits (n<32) and the decimal point position is dynamically controlled for each input image and for each layer (see Non-Patent Document 4). [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Joseph Redmon et.al, “YOLOv3: An Incremental Improvement”, https: / / arxiv.org / abs / 1804.02767 [Non-Patent Document 2] Wei Liu et.al, “SSD: Single Shot MultiBox Detector”, https: / / arxiv.org / pdf / 1512.02325.pdf [Non-Patent Document 3] Zhisheng Li et.al, "Laius: An 8-Bit Fixed-Point CNN Hard ware Inference Engine," 2017 IEEE International Symposium on Parallel and Distributed Processing with Applications and 2017 IEEE International Conference on Ubiquitous Computing and Communications (ISPA / IUCC), Guangzhou, 2017, pp. 143-150, doi: 10.1109 / ISPA / IUCC.2017.00030. [Non-Patent Document 4] A. Hatta, H. Uzawa, S. Yoshida, T. Nitta, “Proposal of a dynamic decimal point position control method for object detection AI inference hardware,” Institute of Electronics, Information and Communication Engineers, September 2020. Summary of the Invention [Problem to be solved by the invention]

[0007] However, the method of dynamically controlling the decimal point position requires calculations to be performed on all pixel values. In addition, calculations are performed continuously even if the same feature map value is repeated. This makes it difficult to reduce power consumption.

[0008] The disclosed technology has been made in consideration of the above points, and aims to provide a data processing device, a data processing method, and a data processing program that can reduce power consumption by skipping operations made possible by increasing the output equivalence rate. [Means for solving the problem]

[0009] A first aspect of the present disclosure is a data processing device configured to set a decimal point position of N-bit (N is a natural number equal to or greater than 2) fixed-length data corresponding to each of a plurality of layers constituting a multilayer neural network, the data processing device being configured to perform arithmetic processing corresponding to each of the plurality of layers constituting the multilayer neural network on each of continuously input processing target data according to a processing algorithm of the multilayer neural network for the N-bit fixed-length data for which the decimal point position has been set by the decimal point position control unit, the data processing device being configured to perform arithmetic processing corresponding to each of the plurality of layers constituting the multilayer neural network, the data processing device being configured to determine equivalent data that increases according to a saturation rate of the processing target data at the time of processing in the arithmetic processing, skip the second and subsequent arithmetic processing of the equivalent data in the arithmetic processing, and to detect the number of detected objects in the processing target data at the time of processing, a frame surrounding the object, an attribute, and metadata including the recognition accuracy of the attribute, and the number of times that an upper limit value (a value when a whole bit string is 1) of a value range determined by the decimal point position is exceeded during the arithmetic processing for each layer of the detection and calculation unit. the decimal point position control unit determines a decimal point position moved for each layer so that the upper limit counter value falls within a first threshold range and the lower limit counter value falls within a second threshold range as a first control, and causes the detection and calculation unit to perform calculation processing on the processing target data at the time of processing; and the saturation rate control unit repeatedly performs calculation processing on the processing target data at the time of processing by the detection and calculation unit. As a second control to be performed, an instruction to move at least the decimal point position is given to the decimal point position control unit as a control to increase a lower limit saturation rate proportional to the magnitude of the counted lower limit counter value with respect to the result of the first control, and the decimal point position control unit performs a predetermined judgment based on the instruction from the saturation rate control unit and the metadata as the second control, determines the decimal point position to which the decimal point has been moved for each layer, and causes the detection calculation unit to perform calculation processing on the processing target data at the time of processing by the detection calculation unit, and the recognition accuracy synthesis unit performs the second control by:The recognition accuracy obtained by the calculation process of the detection calculation unit is replaced with the stored recognition accuracy having a higher accuracy.

[0010] A second aspect of the present disclosure is a data processing method, comprising: setting a decimal point position of N-bit (N is a natural number equal to or greater than 2) fixed-length data corresponding to each of a plurality of layers constituting a multilayer neural network; setting a decimal point position of the fixed-length data; and performing arithmetic processing corresponding to each of the plurality of layers constituting the multilayer neural network on each of continuously input processing target data for the N-bit fixed-length data for which the decimal point position has been set, according to a processing algorithm of the multilayer neural network; in the arithmetic processing, determining equivalent data that increases according to a saturation rate of the processing target data at the time of processing, skipping second and subsequent arithmetic processing of the equivalent data; and in the arithmetic processing, detecting metadata including the number of detected objects in the processing target data at the time of processing, frames surrounding the objects, attributes, and recognition accuracy of the attributes, and in the process of the arithmetic processing for each layer, determining whether or not the number of detected objects is greater than the number of detected objects in the processing target data at the time of processing, frames surrounding the objects, attributes, and recognition accuracy of the attributes, and determining whether or not the number of detected objects is greater than the number of detected objects in the processing target data at the time of processing, according to the decimal point position. the first control counts an upper limit counter value, which is the number of times an upper limit value of a value range determined by the calculation of the upper limit counter value, and a lower limit counter value, which is the number of times the upper limit counter value falls below a lower limit value of the value range; determining a decimal point position to be moved for each layer so that the upper limit counter value falls within a first threshold value range and the lower limit counter value falls within a second threshold value range, and causing a calculation process to be performed on the data to be processed at the time of processing; and issuing an instruction to move at least the decimal point position as a control to increase a lower limit saturation rate proportional to the magnitude of the counted lower limit counter value for a result of the first control; and performing a predetermined judgment based on the instruction and the metadata as the second control, determining the decimal point position to be moved for each layer, and causing a calculation process to be performed on the data to be processed at the time of processing; and replacing the recognition accuracy obtained by the calculation process with the recognition accuracy having a higher accuracy that is stored as the second control. Effect of the Invention

[0011] According to the disclosed technology, power consumption can be reduced by skipping operations, which is made possible by increasing the output equivalence rate. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an object detection device according to a reference example. [Diagram 2] FIG. 13 is a diagram illustrating an example of a CNN in a reference example. [Diagram 3] FIG. 13 is a diagram showing a flow of a convolution calculation in a detection calculation unit of a reference example. [Figure 4] FIG. 13 is a diagram illustrating a flow of a decimal point position control flow of a decimal point position control unit of a reference example. [Diagram 5] FIG. 13 is a diagram illustrating an example of a flow of a process for determining an initial decimal point position in a reference example. [Figure 6] 13 shows an example flow of a process for determining the decimal point position of each layer using the counter of the reference example. [Figure 7] FIG. 2 is a block diagram showing a hardware configuration of an object detection device according to each embodiment of the present disclosure. [Figure 8] FIG. 1 is a block diagram showing a configuration of an object detection device according to each embodiment of the present disclosure. [Figure 9] 13 is a diagram showing an example in which kernel data is shifted by one stride at a time and product-sum calculation processing for one channel is performed in the detection calculation unit according to the embodiment of the present disclosure. FIG. [Figure 10] FIG. 4 is a diagram illustrating an overview of a control method for optimizing a lower limit saturation rate in the first embodiment. [Figure 11] 6 is a diagram showing a decimal point position control flow (lower limit saturation value control flow) of the decimal point position control unit of the first embodiment. FIG. [Figure 12] FIG. 11 is a diagram showing a decimal point position control flow (lower limit saturation value control flow) of the decimal point position control unit of the second embodiment. [Figure 13] FIG. 13 is a diagram showing a decimal point position control flow (lower limit saturation value control flow) of the decimal point position control unit of the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. In addition, the dimensional ratios of the drawings are exaggerated for the convenience of explanation and may differ from the actual ratios.

[0014] First, before describing the embodiments of the present disclosure, the configuration and operation of the technology of Non-Patent Document 4, which is the premise of the technology of the present disclosure, will be described as a reference example. [Reference example] In the reference example, an object detection device corresponding to a data processing device will be described as an example. The object detection device receives and processes a series of multiple input images (hereinafter, may be abbreviated to images) continuously input from a video. The object detection device is a device that performs object detection by detecting metadata including the number of detected objects included in the image, a frame surrounding the object, attributes, and the recognition accuracy of the attributes. Here, the position of the object is represented by the coordinates of the center of the object in the image or a square frame surrounding the object. In addition, the attribute is a type of object such as a person or a car, and may be called a category. In addition, the recognition accuracy is, for example, the probability that a detected object has a specific attribute. Note that a series of multiple input images (images) continuously input from a video is an example of "each of the data to be processed continuously input" of the technology of the present disclosure.

[0015] As shown in FIG. 1, the object detection device 10 according to the reference example includes a detection calculation unit 101, a decimal point position control unit 102, and a counter unit 103. The detection calculation unit 101 is configured to perform calculation processing based on inference processing of deep learning on an input image. The decimal point position control unit 102 determines the position of a decimal point of fixed-length data (hereinafter simply referred to as "decimal point position") to be calculated by the detection calculation unit 101. The counter unit 103 is a collective name for an upper limit counter and a lower limit counter. The counter unit 103 is configured to count the number of times that data exceeds the upper limit and the lower limit of a value range determined by the decimal point position set by the decimal point position control unit 102 during the calculation processing by the detection calculation unit 101, for each layer. The number of times that data exceeds the upper limit of a value range determined by the decimal point position set by the decimal point position control unit 102 and the number of times that data exceeds the upper limit are counted by the counter unit 103.

[0016] The detection calculation unit 101 is, as hardware, a calculation processing circuit configured to perform calculation processing corresponding to each of a plurality of layers constituting a multilayer neural network according to a processing algorithm of the multilayer neural network. The processing by the multilayer neural network in the detection calculation unit 101 typically uses a convolutional neural network (CNN).

[0017] FIG. 2 is a diagram showing an example of a CNN in the reference example. In the CNN, convolution layers (CONV1, CONV2_1, CONV2_2) and pooling layers (Pooling1, Pooling2) that perform a "pooling" process to downsize the result of the convolution process are arranged alternately. The convolution layers convolve a predetermined filter with the input image. The CNN also consists of a "feature extraction part" that creates a feature map, and a "classification part" that identifies objects contained in the input image from the feature map using multiple fully connected layers (FC: Fully Connected Layers; FC1, FC2). In the feature extraction part, a calculation is performed to convolve the filter with the image. In the classification part, a product-sum calculation is performed to multiply the value of each pixel in the feature map by a weight and take the sum, and a bias is added to the result of the product-sum calculation, which is input to an activation function and the output is obtained. In addition, ReLU (Rectified Linear Unit) or the like is used as the activation function. The weight and activation function parameter values ​​can be determined by learning. In addition, in FIG. 2, "BRAM" represents a block RAM that stores the results of calculations in each layer.

[0018] The detection calculation unit 101 uses an object detection algorithm based on deep learning to perform convolution and combination processing of inference processing, and outputs metadata included in the input image. The object detection algorithm is, for example, YOLO (Non-Patent Document 1) or SSD (Non-Patent Document 2).

[0019] The data handled by the detection and calculation unit 101 in the reference example in each layer of the multilayer neural network, for example, the input, output (feature map), weight (kernel), and bias, is fixed-length data having a bit width smaller than 32 bits. The fixed-length data has a data structure that allows a different decimal point position for each layer. More specifically, in the reference example, the detection and calculation unit 101 performs calculation processing corresponding to each of the multiple layers constituting the multilayer neural network on the fixed-length data having a bit width of, for example, 8 bits, and the decimal point position of the handled data is set for each layer by the decimal point position control unit 102 described later.

[0020] FIG. 3 shows the flow of the convolution calculation in the detection calculation unit 101. For example, if the input is 8 bits and the weight is 8 bits, the result obtained by the product-sum calculation will be a maximum of 16 bits. The detection calculation unit 101 adds a bias to the 16-bit calculation result, multiplies it by an activation function, and creates a 16-bit intermediate feature map. Since the feature map will be the input to the next layer, the 16-bit intermediate input map is reduced to an 8-bit width and used as the feature map for the next layer input. Note that the number of layers, the activation function, and the method of adding the bias are appropriately selected for each object detection algorithm used, and do not limit the technology of the reference example.

[0021] [Decimal point position control section] As described above, the decimal point position control unit 102 is a processing circuit that determines the decimal point position of fixed-length data to be calculated in the detection and calculation unit 101, corresponding to each of the multiple layers that make up the multi-layer neural network.

[0022] In the reference example, the decimal point position control unit 102 updates the decimal point position of each layer based on the value of the counter unit 103 and notifies the detection and calculation unit 101. The detection and calculation unit 101 changes the decimal point position of each layer based on the notification. By using the counter unit 103, it is possible to optimize multiple layers at once.

[0023] [Outline of decimal point position control flow] The operation of the object detection device 10 is described by the decimal point position control flow of the decimal point position control unit 102. FIG. 4 is a diagram showing the flow of the decimal point position control flow of the decimal point position control unit 102. The decimal point position control flow in FIG. 4 is the basic flow, and each subroutine will be described later. In the following description, the decimal point position of the fixed-length data set corresponding to the i-th layer may be simply referred to as "the decimal point position of the i-th layer" (where i is an integer greater than 0).

[0024] In the flow of FIG. 4, the process of determining the optimal decimal point position is roughly divided into two steps (Step A and Step C). After the decimal point position is determined, it becomes a process of waiting (Step E, Step F) until there is a significant change in the image type or the like.

[0025] First, initially, an initial value of the decimal point position (initial decimal point position nopt_first) is determined for each layer constituting the multi-layer neural network, and object detection is performed using the fixed-length data of the decimal point position set based on the initial value (Step A). Then, the total number of detected objects (Nd) and detection attributes (Nobj) such as "dog" and "human" obtained as a result of this object detection are saved (Step B). Next, the decimal point position (nopt_i, 0 < i) of each layer is tuned to determine the optimal decimal point position (Step C).

[0026] After tuning the decimal point position of each layer, the decimal point position of each layer is notified to the detection calculation unit 101, object detection processing is performed, and the obtained detection results (Nd, Nobj) are confirmed (Step D). Thereafter, it enters the waiting state for image change (Step E, Step F). If there is no significant change in the image (Step F: NO), the image change check (Step E) is continued. If there is a significant change in the image (Step F: YES), it returns to Step A and restarts the optimization process.

[0027] In the image change check (Step E), image changes may be checked by either (A) a determination process using the total number of detections Nd, or (B) a determination process using the attributes (Nobj) of the detected objects.

[0028] (A) In the judgment process using the total detection count Nd, the total detection count before and after the decimal point position is shifted is compared. The total detection count before the decimal point position is shifted is the total detection count in Step B or Step D in Fig. 4. In the comparison, when the rate of change is 0, it is determined that there is "no change", when the rate of change in the total detection count is equal to or greater than 0 and does not exceed the detection count threshold, it is determined that there is a "small change", and when the rate of change in the total detection count exceeds the detection count threshold, it is determined that there is a "big change".

[0029] (B) In the judgment process using the attributes (Nobj) of the detected objects, when a new attribute is added, the ratio of the number of detected objects with that attribute Nobj to the total number of detections Nd is judged, and “no change”, “small change”, or “big change” is identified.

[0030] The processes of determining the initial decimal point position (nopt_first) (Step A) and determining the decimal point position (nopt_i) of the i-th layer (Step C) in the above-mentioned decimal point position control flow will be described in detail below.

[0031] [Step A: Determine the initial decimal point position (nopt_first)] Fig. 5 shows a flow chart of an example of a process for determining the initial decimal point position (nopt_first). In this determination method, the decimal point position is changed while the decimal point position is made common to all layers, so that the range of fixed-length data and the minimum step size used in the calculation process corresponding to each layer are made common to detect objects. The decimal point position n that gives the best detection result is then set as the initial decimal point position not_first common to all layers.

[0032] Specifically, object detection is performed for each input image while sequentially changing the decimal point position n of all layers from an initial value ninit to a maximum value nmax, and the total number of detections Nd and the recognition accuracy A are saved for each decimal point position (Step A1). Then, from the total number of detections Nd and the recognition accuracy A obtained for each decimal point position, the recognition accuracy A for each detected object, i.e., the average value of the recognition accuracy A (ΣA / Nd), is calculated. The decimal point position n at which the average value of the recognition accuracy A is the largest is set as the initial decimal point position nopt_first (Step A2). Note that the initial value ninit and the maximum value nmax are determined by the device designer and do not limit the technology of the reference example.

[0033] Also, as an example of a method for determining the initial decimal point position (nopt_first), a method for determining an initial value using multiple input images has been described, but the reference example is not limited to this. For example, an object detection simulator or the like may be used in advance to analyze the value range of a feature map, which is an output result for multiple images, and the initial value may be determined based on the analysis results. Also, in the reference example, the definition of "decimal point position with the best detection result" is "decimal point position with the highest average recognition accuracy," but the definition is not limited to this. For example, "decimal point position with the highest number of detections" may be defined as "decimal point position with the best detection result."

[0034] [Step C: Determining the decimal point position (nopt_i) of the i-th layer] 6 shows a flow chart of an example of a process for determining the decimal point position (nopt_i) of each layer using counter unit 103. Here, the values ​​of counter unit 103 are an upper limit counter value and a lower limit counter value. The upper limit counter value is the number of times that the fixed length data handled in each layer exceeds the upper limit value of the range determined by the decimal point position of that layer (the value when the entire bit string is 1). The lower limit counter value is the number of times that the fixed length data handled in each layer falls below the lower limit value of the range determined by the decimal point position of that layer (the value when only the LSB (least significant bit) is 1 of the entire bit string).

[0035] In Step C, first, a layer is identified in which the upper limit counter value and the lower limit counter value are smaller than the first threshold UPth and the second threshold UNth (UPth≧0, UNth≧0) as a result of object detection using the initial value ninit. This layer is regarded as a "layer with an optimized decimal point position," and the decimal point position of this layer is left unchanged. In the next Step C2a, decimal point position control is performed on layers other than this layer (Step C1a).

[0036] At this time, the thresholds UPth and UNth (≧0) are set to values ​​that can be set for each layer. In Step C2a of Fig. 6, the decimal point position of the "non-optimized layers" is changed. Specifically, the upper limit counter value and the lower limit counter value of these layers are set to fall within the above-mentioned conditions. That is, the decimal point position of the non-optimized layers is changed each time an image is input so that the upper limit counter value is less than the first threshold UPth and the lower limit counter value is less than the second threshold UNth.

[0037] For example, first, the upper limit value is focused on, and the decimal point position is changed so that the upper limit counter value is smaller than the first threshold UPth. The upper limit value is a value exceeding the first threshold UPth, and the lower limit value is a value exceeding the second threshold UNth. After the upper limit counter value is optimized, the lower limit counter value is changed so that it is smaller than the second threshold UNth. When the upper limit counter value and the lower limit counter value converge, the decimal point position is set as the optimal position nopt_i and the process transitions to Step D shown in FIG. 4. For layers that are only below one of the thresholds as a result of control based on the upper limit counter value and the lower limit counter value, it is determined in advance which condition (smaller than UPth or smaller than UNth) to prioritize for each layer, and the case where the prioritized condition is met is set as the "optimized decimal point position."

[0038] Also, the decimal point position may be changed by combining the value of the counter unit 103 and a method using metadata, using metadata output from the detection calculation unit 101. When detecting an object in a video, the objects in the continuously input images change little by little, and it is rare for all of them to change in a short period of time. Therefore, in the method using metadata, the decimal point position of each layer is not controlled from the detection result of only one input image, but the decimal point position of each layer is calculated repeatedly using the object detection results (metadata) for multiple input images. This allows the decimal point position of each layer to be optimized little by little.

[0039] When combining the counter unit 103 and metadata, the detection results (A, Nd) obtained in the process of optimizing the decimal point position using the value of the counter unit 103 are stored, and the decimal point position that provides the best result is selected from the stored results.

[0040] [Overview of the embodiment of the present disclosure] The method of the embodiment of the present disclosure is a method made to solve the problems in Non-Patent Document 4 and the Reference Example. The purpose is to realize low power consumption while maintaining recognition accuracy by setting data used in calculations to n bits in order to achieve circuit miniaturization and controlling the decimal point position for each layer and each image.

[0041] Therefore, in this disclosure, when the calculation result exceeds the maximum value (i.e., upper limit saturation occurs) or falls below the minimum value (i.e., lower limit saturation occurs), the results that exceed and fall below the minimum value all become the same data as the maximum and minimum values. If saturation can be intentionally generated, the equality rate of the calculation results will be high. The equality rate is the rate of the results that are equal to the total number of pixels. In addition, by utilizing the fact that the lower limit value, which has a small absolute value of the calculation result, does not greatly affect the recognition accuracy, the equality rate of the feature map (output from the previous layer) is improved by increasing the lower limit saturation rate, and the calculation skip is executed. The lower limit saturation rate is the rate of data whose lower limit value is saturated (exceeding the lower limit threshold) among the data of the feature map to be calculated. Therefore, the lower limit saturation rate is proportional to the size of the lower limit counter value that is counted. In other words, the equality rate can be increased by increasing the lower limit saturation rate. The equality rate is the rate at which the calculated product-sum calculation results become equal data. In addition, low power consumption is achieved by stopping the clock of the circuit during the execution of the calculation skip. Furthermore, although an improvement in the equivalence rate may cause a deterioration in recognition accuracy, in order to compensate for the deterioration, a process is performed in which the recognition accuracy result is replaced with the recognition accuracy result before the lower limit saturation rate was increased.

[0042] As a result, the rate of identical values ​​in the feature map is increased while suppressing the deterioration of recognition accuracy, and power consumption is reduced by skipping calculations.In addition, by increasing the rate of identical values ​​in the feature map, it becomes possible to use the results of calculations that have been calculated once without recalculating parts with consecutive identical values, which makes it possible to improve the average calculation speed.

[0043] [First embodiment of the present disclosure] The configuration and operation of the first embodiment of the present disclosure will be described below based on the reference example. Note that processing units with the same names as those in the reference example will not be described as they can perform similar processing.

[0044] 7 is a block diagram showing a hardware configuration of an object detection device 20 according to each embodiment of the present disclosure. The object detection device 20 is an example of a data processing device of the technology of the present disclosure.

[0045] 7, the object detection device 20 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0046] The CPU 11 is a central processing unit, and executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various arithmetic processing according to the program stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores an object detection program for executing various processes of the object detection device 20. The object detection program is an example of a data processing program of the technology disclosed herein.

[0047] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system, and various data.

[0048] The input unit 15 includes a pointing device such as a mouse, and a keyboard, and is used to perform various input operations.

[0049] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may function as the input unit 15 by adopting a touch panel system.

[0050] The communication interface 17 is an interface for communicating with other devices such as terminals. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0051] Next, a description will be given of each functional configuration of the object detection device 20. Fig. 8 is a block diagram showing the configuration of the object detection device 20 according to each embodiment of the present disclosure. Each functional configuration is realized by the CPU 11 reading out an object detection program stored in the ROM 12 or the storage 14, expanding it in the RAM 13, and executing it.

[0052] 8, the object detection device 20 includes a detection calculation unit 201, a decimal point position control unit 202, a counter unit 203, a saturation rate control unit 204, and a recognition accuracy synthesis unit 205. Since the detection calculation unit 201 executes many product-sum operations, it is often realized using a multi-core CPU or a GPU (Graphics Processing Unit). The detection calculation unit 201 may be realized, for example, by an FPGA (Field Programmable Gate Array).

[0053] The detection calculation unit 201 is a calculation processing circuit capable of performing calculation processing similar to that of the detection calculation unit 101 of the reference example. The detection calculation unit 201 determines equivalent data that increases according to the saturation rate of the image at the time of processing in the calculation processing, and skips the second and subsequent calculation processing of the equivalent data. By skipping the calculation in this manner, the processing of the detection calculation unit 201 takes the equivalent rate into consideration. Also, in the calculation processing, metadata (detection results) including the number of objects detected in the image at the time of processing, the frame surrounding the object, attributes, and the recognition accuracy of the attributes are detected. The metadata is used in the process of optimizing the lower limit saturation rate using the saturation rate control unit 204 described later.

[0054] In addition to the basic convolution operation shown in the reference example, the detection operation unit 201 performs operation skip in units of kernel size and clock gate during skip. FIG. 9 shows an example of performing multiplication and accumulation operation processing for one channel by shifting 5×5 input data and 3×3 kernel data by one stride at a time. Shifting by one slide means sliding the kernel size by one unit. The input data is a feature map of the output obtained in the previous layer. The area surrounded by the input value thick frame in FIG. 9 is the first input value, and the area surrounded by the input value thick dotted line frame is the second input value. The operation skip in units of kernel size is an operation skip that is possible in units of kernel size when the first input values ​​corresponding to the size of the 3×3 kernel are all the same value and the second input values ​​to be the next operation target shifted by one stride are also all the same value. FIG. 9 shows that if the first operation result is saved, the operation can be skipped in the second operation by using the saved value. First, it is determined whether or not there is an equal value in units of kernel size, and if all values ​​are the same, the product-and-accumulation operation is performed and the value is stored. Next, when the kernel is shifted, it is determined whether the shifted values ​​are all the same value. If they are the same value, the stored calculation result is used, and the clock is stopped during that time to reduce power consumption. For example, when performing a multiplication and accumulation operation for 1ch with 1 stride on 5×5 input data and 3×3 kernel data shown in FIG. 9, all of the input data up to the 4th row are the same value (I00). Therefore, f00 (=I00×(k00+k01+k02+k10+k11+k12+k20+k21+k22)), which is the first calculation result, is the same value up to the 2nd row of the multiplication and accumulation operation result. In this example, out of the total 6 calculations up to the 2nd row, the 2nd to 6th calculations except the 1st can be skipped. Note that the number of layers, activation function, and bias addition method differ depending on the object detection algorithm used, and are not specified in this disclosure.

[0055] Control is performed in cooperation between the decimal point position control unit 202 and the saturation rate control unit 204. The process of optimizing the lower limit saturation rate through cooperation will be described in detail in the explanation of the operation, and only the functional configuration will be described here.

[0056] The decimal point position control unit 202 performs a first control to determine the decimal point position and perform calculation processing by the same process as in the above reference example (this is the process of the 1st step described later). As the first control, the decimal point position control unit 202 determines the decimal point position by moving the decimal point position for each layer so that the upper limit counter value falls within the range of the first threshold value and the lower limit counter value falls within the range of the second threshold value. Then, as the first control, calculation processing is performed on the image at the time of processing by the detection calculation unit 201.

[0057] Moreover, the decimal point position control unit 202 optimizes the lower limit saturation rate by repeatedly performing the second control (this is the processing of the 2nd step described later). The decimal point position control unit 202 performs a predetermined judgment based on an instruction from the saturation rate control unit 204 and metadata, determines the decimal point position to which the decimal point position for each layer has been moved, and causes the detection calculation unit 201 to perform calculation processing on the image at the time of processing.

[0058] The counter unit 203 has the same functional configuration as the counter unit 103 of the reference example. The counter unit 203 counts an upper limit counter value, which is the number of times an upper limit value of a range determined by the decimal point position is exceeded, and a lower limit counter value, which is the number of times a lower limit value of a range is exceeded, during the process of calculation processing for each layer by the detection calculation unit 201.

[0059] Saturation rate control unit 204 operates according to instructions from decimal point position control unit 202. The instruction here may be to accept the result of the first control. Saturation rate control unit 204 performs control to increase the lower limit saturation rate in response to the result of the first control. As a control, an instruction to move at least the decimal point position is performed on decimal point position control unit 202, and the specific contents of the control will be explained in the action section.

[0060] Depending on the judgment of the decimal point position control unit 202, the recognition accuracy synthesis unit 205 replaces the recognition accuracy in the second control obtained by the calculation processing of the detection calculation unit 201 with the recognition accuracy in the first control which is stored and has higher accuracy, and outputs it as the detection result.

[0061] Before describing the operation, a control method for optimizing the lower limit saturation rate will be described. FIG. 10 is a diagram showing an overview of a control method for optimizing the lower limit saturation rate in the first embodiment. The control flow is roughly divided into two steps. In the 1st step, a threshold is used to first determine an appropriate decimal point position for each layer (standard decimal point position determination). In the 2nd step, the lower limit second threshold is gradually increased by ΔUNth so that the lower limit saturation rate increases from the decimal point position determined in the 1st step (improvement of lower limit saturation rate). This increases the lower limit saturation rate as much as possible, and the decimal point position at which the maximum saturation rate is obtained is operated as a steady state (equivalence rate steady section). Then, if a change occurs in the detection result, the process returns to the 1st step.

[0062] Next, the operation of the object detection device 20 will be described. The operation of the object detection device 20 of this embodiment will be described by the decimal point position control flow of the decimal point position control unit 202, similar to the reference example. The CPU 11 reads out the object detection program from the ROM 12 or the storage 14, expands it in the RAM 13, and executes it, thereby processing the decimal point position control flow. The decimal point position control flow in the operation is an example of a data processing method of the technology of the present disclosure.

[0063] [Decimal point position control flow in the first embodiment] 11 is a diagram showing a decimal point position control flow (lower limit saturation value control flow) of the decimal point position control unit 202 of the first embodiment. The processes of steps S100 and S102 are an example of the first control of the technique of the present disclosure, and the processes of the subsequent steps are an example of the second control. In the decimal point position control flow, the part of the second control is the lower limit saturation value control flow. In the following, the symbols attached to terms such as the first threshold value and the second threshold value will be indicated only when necessary, and will be omitted as appropriate.

[0064] In step S100, the CPU 11 determines the decimal point position to which the upper limit counter value is moved for each layer so that the upper limit counter value falls within the range of the first threshold value and the lower limit counter value falls within the range of the second threshold value. The process of this step is executed by cooperation between the decimal point position control unit 202 and the counter unit 203. Note that the process of this step can be realized by the same process as the processes from Step A to Step C in the reference example. The process of this embodiment corresponding to Step C in the reference example will be described below.

[0065] The upper limit counter value and the lower limit counter value are counter values ​​that measure the number of times that the calculation result of each layer goes beyond the absolute value of the value range determined by the data bit width and the decimal point position. When these counter values ​​become large, the upper limit value, the lower limit value, or the number of values ​​that are rounded to 0 increases, and the recognition accuracy decreases. In order to reduce these counter values, the decimal point position is moved, but since the data bit width is limited, it is difficult to make the counter value = 0 in all layers even if the decimal point position is changed. Therefore, by setting an initial first threshold UPth_first and a second threshold UNth_first for each layer and controlling the decimal point position so that the counter value falls within the thresholds, it becomes possible to detect objects without degrading the recognition accuracy.

[0066] The upper and lower counter values ​​obtained by object detection are compared with the first and second thresholds, respectively, and the standard decimal point position is determined by changing the decimal point position for each layer as shown in (A) to (C) below.

[0067] (A) If the upper limit counter value < first threshold UPth_first and the lower limit counter value < second threshold UNth_first, the decimal point position is not changed and is left as is. (B) If the upper limit counter value ≥ first threshold UPth_first, the decimal point position is moved by 1 towards the LSB. (C) If the upper limit counter value = 0 and the lower limit counter value ≥ UNth_first, the decimal point position is moved by 1 towards the MSB (most significant bit). This operation is repeated for multiple images, and the decimal point position is changed so that the upper limit counter value falls within the first threshold and the lower limit counter value falls within the second threshold. For layers that do not fall within the thresholds, the upper limit counter value is controlled so that it falls within the first threshold.

[0068] In step S102, the CPU 11 performs object detection processing using the decimal point position determined in step S102, and determines and stores the number of detections Nd, the attribute Nobj, a frame indicating the object position, and the recognition accuracy as the obtained detection results.

[0069] Next, in the second step, the lower limit saturation rate is increased within a range where the number of detected objects does not change.

[0070] In step S104, the CPU 11, functioning as the decimal point position control unit 202, issues an instruction to the saturation rate control unit 204 to increase the second threshold value UNth of the predetermined layer by ΔUNth (UNth=UNth+ΔUNth). Here, the predetermined layer is all layers, but may be a specified layer that has been determined in advance. In step S106, the CPU 11 increases the second threshold UNth, changes the decimal point position so that the counter value falls within the threshold, and performs object detection. Here, object detection is performed twice using two input images, and the decimal point position is changed in the object detection for each image as follows. (1): The decimal point position of all layers is moved by 1 to the LSB side, and object detection is performed. The movement to the LSB side is in the direction of increasing the lower limit saturation rate. (2): For layers in which the lower limit counter value obtained as a result of detection in (1) becomes larger than the second threshold UNth, the value is moved by 1 to the MSB side to return to the original value, and object detection is performed. Note that the number of detections Nd, attribute Nobj, frame indicating the position of the object, and recognition accuracy obtained as detection results are saved. The processing of this step is performed by cooperation between the decimal point position control unit 202 and the detection calculation unit 201.

[0071] In step S108, CPU 11 compares the number of detections and the attributes from the detection results in (2) of step S106 with those stored in step S102 to determine whether there has been a change. If there has been no change, the process proceeds to step S110, and if there has been a change, the process proceeds to step S120.

[0072] In step S110, CPU 11 determines whether the overlapping area between the frame indicating the object position obtained in step S102 and the frame indicating the object position obtained in step S106 is equal to or greater than the position threshold. The overlapping area indicates the area of ​​overlapping when the frames indicating the object position obtained in each of the two object detections are overlapped. If the overlapping area is equal to or greater than the position threshold, it is determined that there is no image change and that the second threshold has been increased to a range where no false detection occurs, and the process proceeds to step S112. If the overlapping area is not equal to or greater than the position threshold, it is determined that there has been an image change and the process proceeds to step S120.

[0073] In step S112, the CPU 11 replaces the recognition accuracy of each object with the recognition accuracy of the first control (step S102) and outputs it as a detection result. This process is executed by the decimal point position control unit 202 issuing an instruction to the recognition accuracy synthesis unit 205 to replace the accuracy. After this step is completed, the process returns to step S104, and this is repeated until the number of detections and the attribute change. This performs the process of improving the lower limit saturation rate shown in FIG. 10.

[0074] In step S120, the CPU 11 replaces the recognition accuracy of each object with the recognition accuracy of the first control (step S102), and outputs the obtained recognition accuracy for the new object as the detection result. In this way, the recognition accuracy for the object that continues to be detected is maintained.

[0075] In step S122, the CPU 11, functioning as the decimal point position control unit 202, issues an instruction to the saturation rate control unit 204 to decrease the second threshold value UNth of a predetermined layer by ΔUNth (UNth=UNth−ΔUNth).

[0076] In step S124, the CPU 11 reduces the second threshold UNth, changes the decimal point position so that the counter value falls within the threshold, and performs object detection. The number of detections Nd, attribute Nobj, a frame indicating the object position, and recognition accuracy obtained as detection results are saved. The processing of this step is performed by the cooperation of the decimal point position control unit 202 and the detection calculation unit 201.

[0077] In step S126, CPU 11 compares the number of detections and the attribute in step S124 with those stored in step S102, and judges whether the number of detections and the attribute are the same. If they are the same, the process proceeds to step S128, and if they are not the same, the process returns to the process of step S100. If they are the same, it can be judged that the previous judgment result in step S108 is a change due to the occurrence of a false detection, and that the recognition accuracy will be maintained if the second threshold value in the previous stage is used.

[0078] In step S128, the CPU 11 determines to use the second threshold value decreased in step S122 as the steady threshold value. This causes object detection to be performed using the steady threshold value, resulting in an improvement in the lower limit saturation rate in object detection and an improvement in the equivalence rate.

[0079] In step S130, the CPU 11 performs an image change check. The image change check may be performed, for example, by determining the number of detections and attributes as in the reference example. In addition, the image change check is not limited to this, and may be performed by using a timeout, for example, to treat the image as having changed after a certain period of time has elapsed.

[0080] In step S132, if the image change check indicates that the image change is large, the CPU 11 returns to the process of step S100, and if the image change is not large, the CPU 11 repeats step S130 and this step. As a result, it is possible to increase the lower limit saturation rate without deteriorating the recognition accuracy.

[0081] As shown in the above operation, the saturation rate control unit 204 instructs the second threshold UNth of the predetermined layer to be increased by ΔUNth (step S104). The decimal point position control unit 202 also causes the detection calculation unit 201 to perform calculation processing based on the second threshold UNth increased by the instruction (step S104), and performs a first determination to determine whether the metadata in the calculation result has changed with respect to the first control (steps S106 to S110). If it is determined that there has been no change, the second threshold is repeatedly increased (steps S104 to S112). If it is determined that there has been a change in the first determination, the saturation rate control unit 204 also instructs the second threshold of the predetermined layer to be decreased (steps S120 and S122). The decimal point position control unit 202 also causes the detection calculation unit 201 to perform calculation processing based on the second threshold UNth decreased by the instruction, and performs a second determination to determine whether the metadata in the calculation result has changed with respect to the first control (step S124). If it is determined in the second determination that there has been no change, the second threshold value is set as the steady threshold value, and if it is determined that there has been a change, the process returns to the first control (step S126).

[0082] As described above, according to the object detection device 20 of this embodiment, power consumption can be reduced by skipping calculations, which is made possible by increasing the output equality rate.

[0083] [Second embodiment of the present disclosure] The second embodiment is an aspect in which the lower limit saturation rate is improved by moving the decimal point position without changing the second threshold. In the first embodiment, the method is to increase the pixel data that reaches saturation by increasing the second threshold after determining the optimal decimal point position. In this case, it is necessary to use at least two input images because the decimal point position is determined after increasing the second threshold. The difference between the second embodiment and the first embodiment is that the method of improving the equivalence rate in the 2nd step is to directly change the decimal point position instead of increasing the threshold. In the second embodiment, a method of improving the equivalence rate with one input image is shown. Note that the configuration of the second embodiment is the same as that of the first embodiment, so the explanation is omitted and only the difference in processing in the action is explained.

[0084] 12 shows a decimal point position control flow (lower limit saturation value control flow) in the second embodiment. The difference from the first embodiment is that step S104 and step S124 are not included.

[0085] In the second embodiment, the process proceeds to step S200 after step S102. In step S200, the CPU 11 moves the decimal point position of a predetermined layer by 1 toward the LSB side, performs object detection, and stores the detection result. After step S200, the process proceeds to step S108. Also, after step S120, the process proceeds to step S202. In step S202, the CPU 11 moves the decimal point position of a predetermined layer by 1 toward the MSB side, performs object detection, and stores the detection result. After step S202, the process proceeds to step S126. As a result of the above, it is possible to increase the saturation rate in a shorter time, and to reduce the average power when observing for a long time.

[0086] As shown in the above operation, the saturation rate control unit 204 instructs the detection calculation unit 201 to move the decimal point position of the predetermined layer by a bits (a>0) to the least significant bit side in the instruction (step S200). The decimal point position control unit 202 also causes the detection calculation unit 201 to perform calculation processing using the decimal point position after the movement to the least significant bit side based on the instruction (step S200). Also, a first judgment is performed to judge whether the metadata in the calculation result has changed with respect to the first control (steps S106 to S110). When it is judged that the saturation rate control unit 204 has changed in the first judgment, the saturation rate control unit 204 instructs the detection calculation unit 201 to move the decimal point position of the predetermined layer by a bits (a>0) to the most significant bit side (step S202). Also, the decimal point position control unit 202 causes the detection calculation unit 201 to perform calculation processing using the decimal point position after the movement to the most significant bit side based on and performs a second judgment to judge whether the metadata in the calculation result has changed with respect to the first control (step S124). If it is determined in the second determination that there has been no change, the second threshold value is set as the steady threshold value, and if it is determined that there has been a change, the process returns to the first control (step S126).

[0087] As described above, according to the object detection device 20 of this embodiment, the process of improving the equivalence rate can be made more efficient.

[0088] [Third embodiment of the present disclosure] In the third embodiment, even during the process of increasing the equivalence rate, object detection is performed at an optimal decimal point position at regular intervals, and the results are overwritten. In the first and second embodiments, the recognition accuracy (step S112) replaced while the equivalence rate is being increased is the recognition accuracy when detection is performed at the decimal point position determined in the first control. However, in this case, even if the number of objects to be actually detected increases in the image at the time of processing during the process of increasing the equivalence rate, there is a risk that some objects will not be detected due to the increase in the equivalence rate. Therefore, in this embodiment, even during the process of increasing the equivalence rate, object detection is performed at an optimal decimal point position at regular intervals, and the results are overwritten.

[0089] 13 shows a decimal point position control flow (lower limit saturation value control flow) in the third embodiment. The difference from the second embodiment is that the process proceeds to step S300 after step S112.

[0090] In step S300, it is determined whether or not a certain period of time has passed. If the certain period of time has passed, the process proceeds to step S302, and if the certain period of time has not passed, the process returns to step S104.

[0091] In step S302, the CPU 11 changes the decimal point position to the one used in step S102, i.e., the decimal point position determined in the first control, performs object detection, and overwrites and saves the detection result, thereby updating the detection result of step S102.

[0092] In step S304, the CPU 11 returns the decimal point position to the position to which it was moved in step S200, and the process returns to step S200.

[0093] By the above operation, the decimal point position control unit 202, after making a predetermined judgment, judges whether or not a predetermined condition is satisfied (step S300). The decimal point position control unit 202 causes the detection calculation unit 201 to obtain metadata again for the processing target data at the time of the calculation process, using the decimal point position determined in the first control. The recognition accuracy synthesis unit 205 overwrites the metadata obtained in the first control with the re-obtained metadata (step S302).

[0094] As described above, according to the object detection device 20 of this embodiment, object detection can be performed without deteriorating accuracy even in the process of improving the equivalence rate.

[0095] In addition, various processes executed by the CPU by reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. In this case, examples of the processor include a dedicated electric circuit that is a processor having a circuit configuration designed exclusively for executing a specific process, such as a programmable logic device (PLD) whose circuit configuration can be changed after manufacturing an FPGA, and an application specific integrated circuit (ASIC). In addition, various processes may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, and a combination of a CPU and an FPGA). In addition, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0096] In addition, in each of the above embodiments, the object detection program is described as being pre-stored (installed) in the storage 14, but the present invention is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0097] The following supplementary notes are further disclosed regarding the above embodiment.

[0098] (Additional note 1) Memory, at least one processor coupled to the memory; Including, The processor, The method is configured to set a decimal point position of N-bit (N is a natural number of 2 or more) fixed-length data corresponding to each of a plurality of layers constituting the multilayer neural network, and set the decimal point position of the fixed-length data; the multi-layer neural network is configured to perform arithmetic processing corresponding to each of the plurality of layers constituting the multi-layer neural network on each of the continuously input processing target data for the N-bit fixed-length data in which the decimal point position is set, in accordance with a processing algorithm of the multi-layer neural network, in which in the arithmetic processing, equal-value data which increases according to a saturation rate of the processing target data at the time of processing is determined, and the second and subsequent arithmetic processing of the equal-value data is skipped, and in the arithmetic processing, metadata including the number of objects detected in the processing target data at the time of processing, frames surrounding the objects, attributes, and recognition accuracy of the attributes is detected; In the process of the calculation process for each layer, an upper limit counter value is counted as the number of times an upper limit value of a range determined by the decimal point position is exceeded, and a lower limit counter value is counted as the number of times a lower limit value of the range is exceeded, as a first control, a decimal point position is determined for each layer so that the upper limit counter value falls within a first threshold range and the lower limit counter value falls within a second threshold range, and an arithmetic process is performed on the processing target data at the processing time; a second control to be repeated, which is a control for increasing a lower limit saturation rate proportional to the magnitude of the lower limit counter value to be counted as a result of the first control, includes an instruction to move at least the decimal point position; As the second control, a predetermined judgment is made based on the instruction and the metadata, the decimal point position to be moved for each layer is determined, and an arithmetic process is performed on the processing target data at the processing time; As the second control, the recognition accuracy obtained by the calculation process is replaced with a stored recognition accuracy having a higher accuracy. 2. A data processing device configured as follows:

[0099] (Additional note 2) A non-transitory storage medium storing a program executable by a computer to perform data processing, The method is configured to set a decimal point position of N-bit (N is a natural number of 2 or more) fixed-length data corresponding to each of a plurality of layers constituting the multilayer neural network, and set the decimal point position of the fixed-length data; the multi-layer neural network is configured to perform arithmetic processing corresponding to each of the plurality of layers constituting the multi-layer neural network on each of the continuously input processing target data for the N-bit fixed-length data in which the decimal point position is set, in accordance with a processing algorithm of the multi-layer neural network, in which in the arithmetic processing, equal-value data which increases according to a saturation rate of the processing target data at the time of processing is determined, and the second and subsequent arithmetic processing of the equal-value data is skipped, and in the arithmetic processing, metadata including the number of objects detected in the processing target data at the time of processing, frames surrounding the objects, attributes, and recognition accuracy of the attributes is detected; In the process of the calculation process for each layer, an upper limit counter value is counted as the number of times an upper limit value of a range determined by the decimal point position is exceeded, and a lower limit counter value is counted as the number of times a lower limit value of the range is exceeded, as a first control, a decimal point position is determined for each layer so that the upper limit counter value falls within a first threshold range and the lower limit counter value falls within a second threshold range, and an arithmetic process is performed on the processing target data at the processing time; a second control to be repeated, which is a control for increasing a lower limit saturation rate proportional to the magnitude of the lower limit counter value to be counted as a result of the first control, includes an instruction to move at least the decimal point position; As the second control, a predetermined judgment is made based on the instruction and the metadata, the decimal point position to be moved for each layer is determined, and an arithmetic process is performed on the processing target data at the processing time; As the second control, the recognition accuracy obtained by the calculation process is replaced with a stored recognition accuracy having a higher accuracy. Non-transitory storage medium. [Explanation of symbols]

[0100] 20 Object detection device 201 Detection and calculation unit 202 Decimal point position control unit 203 Counter section 204 Saturation rate control section 205 Recognition accuracy synthesis section

Claims

1. a decimal point position control unit configured to set a decimal point position of N-bit (N is a natural number of 2 or more) fixed-length data corresponding to each of a plurality of layers constituting the multilayer neural network, the decimal point position control unit setting the decimal point position of the fixed-length data; a detection and calculation unit configured to perform arithmetic processing corresponding to each of the plurality of layers constituting the multilayer neural network on each of the continuously input processing target data in accordance with a processing algorithm of the multilayer neural network for the N-bit fixed length data in which the decimal point position has been set by the decimal point position control unit, the detection and calculation unit determining equivalent data that increases in accordance with a saturation rate of the processing target data at the time of processing in the arithmetic processing, skipping the second and subsequent arithmetic processing of the equivalent data in the arithmetic processing, and detecting metadata including the number of objects detected in the processing target data at the time of processing, frames surrounding the objects, attributes, and recognition accuracy of the attributes; a counter unit that counts an upper limit counter value, which is the number of times an upper limit value of a value range determined by the decimal point position is exceeded, and a lower limit counter value, which is the number of times a lower limit value of the value range is exceeded, during the process of calculation processing for each layer of the detection calculation unit; A saturation rate control unit and a recognition accuracy synthesis unit are included, the decimal point position control unit determines a decimal point position moved for each layer so that the upper limit counter value falls within a first threshold range and the lower limit counter value falls within a second threshold range, and causes the detection and calculation unit to perform a calculation process on the processing target data at the time of processing; The saturation rate control unit instructs the decimal point position control unit to increase the second threshold value of a predetermined layer as a second control repeated by the decimal point position control unit; the decimal point position control unit causes the detection and calculation unit to perform a calculation process based on the increased second threshold as the second control, performs a first determination to determine a change in the metadata in the calculation result relative to the first control, and when it is determined in the first determination that there is no change, repeats increasing the second threshold; The recognition accuracy combination unit performs the second control by replacing the recognition accuracy obtained by the calculation process of the detection calculation unit with a recognition accuracy having a higher accuracy stored in the memory, the saturation rate control unit instructs the second threshold value of a predetermined layer to be decreased when it is determined that the first determination has been made, the decimal point position control unit causes the detection and calculation unit to perform calculation processing based on the reduced second threshold, performs a second determination to determine a change in the metadata in the calculation result relative to the first control, and when it is determined in the second determination that there is no change, sets the second threshold as a steady threshold, and when it is determined that there is a change, returns to the first control. Data processing device.

2. A decimal point position control unit configured to set the decimal point position of N-bit (N is a natural number of 2 or more) fixed-length data corresponding to each of a plurality of layers constituting a multilayer neural network, said decimal point position control unit setting the decimal point position of said fixed-length data; a detection and calculation unit configured to perform arithmetic processing corresponding to each of the plurality of layers constituting the multilayer neural network on each of the continuously input processing target data in accordance with a processing algorithm of the multilayer neural network for the N-bit fixed length data in which the decimal point position has been set by the decimal point position control unit, the detection and calculation unit determining equivalent data that increases in accordance with a saturation rate of the processing target data at the time of processing in the arithmetic processing, skipping the second and subsequent arithmetic processing of the equivalent data in the arithmetic processing, and detecting metadata including the number of objects detected in the processing target data at the time of processing, frames surrounding the objects, attributes, and recognition accuracy of the attributes; a counter unit that counts an upper limit counter value, which is the number of times an upper limit value of a value range determined by the decimal point position is exceeded, and a lower limit counter value, which is the number of times a lower limit value of the value range is exceeded, during the process of calculation processing for each layer of the detection calculation unit; A saturation rate control unit and a recognition accuracy synthesis unit are included, the decimal point position control unit determines a decimal point position moved for each layer so that the upper limit counter value falls within a first threshold range and the lower limit counter value falls within a second threshold range, and causes the detection and calculation unit to perform a calculation process on the processing target data at the time of processing; the saturation rate control unit instructs the decimal point position control unit to move the decimal point position of a predetermined layer by a bits (a>0) toward the least significant bit side as a second control repeated by the decimal point position control unit; the decimal point position control unit causes the detection and calculation unit to perform a calculation process using the decimal point position after the second control, performs a first determination to determine a change in the metadata in the calculation result relative to the first control, and when it is determined in the first determination that there is no change, repeats the movement toward the least significant bit; The recognition accuracy combination unit replaces the recognition accuracy obtained by the calculation process of the detection calculation unit with a stored recognition accuracy having a higher accuracy as the second control, the saturation rate control unit instructs, when it is determined in the first determination that there is a change, to move the decimal point position of a predetermined layer by a bits (a>0) toward the most significant bit side; the decimal point position control unit causes the detection and calculation unit to perform a calculation process using the decimal point position after the decimal point position has been moved to the most significant bit side, performs a second determination to determine a change in the metadata in the calculation result relative to the first control, and when it is determined in the second determination that there has been no change, sets the second threshold value as a steady threshold value, and when it is determined that there has been a change, returns to the first control. Data processing device.

3. the decimal point position control unit, when a predetermined condition is satisfied after performing the first determination, causes the detection and calculation unit to obtain again the metadata for the processing target data at the time of the calculation process, using the decimal point position determined in the first control; 3. The data processing device according to claim 1, wherein the recognition accuracy synthesis unit overwrites the metadata obtained in the first control with the metadata obtained again.

4. 4. The data processing device according to claim 1, wherein the detection and calculation unit stops a clock for a circuit while executing the skip of the calculation process.

5. The multi-layer neural network is configured to set a decimal point position of N-bit (N is a natural number of 2 or more) fixed-length data corresponding to each of a plurality of layers constituting the multi-layer neural network, and the decimal point position of the fixed-length data is set; the multi-layer neural network is configured to perform arithmetic processing corresponding to each of the plurality of layers constituting the multi-layer neural network on each of the continuously input processing target data in accordance with a processing algorithm of the multi-layer neural network for the N-bit fixed-length data in which the decimal point position is set, the arithmetic processing determining unit 104 determines equivalent data which increases according to a saturation rate of the processing target data at the time of processing in the arithmetic processing, skips the second and subsequent arithmetic processing of the equivalent data, and detects metadata including the number of objects detected in the processing target data at the time of processing in the arithmetic processing, frames surrounding the objects, attributes, and recognition accuracy of the attributes; In the process of the calculation process for each layer, an upper limit counter value is counted as the number of times an upper limit value of a range determined by the decimal point position is exceeded, and a lower limit counter value is counted as the number of times a lower limit value of the range is exceeded, as a first control, a decimal point position is determined for each layer so that the upper limit counter value falls within a range of a first threshold value and the lower limit counter value falls within a range of a second threshold value, and an arithmetic process is performed on the processing target data at the processing time; Instructing the second control to be repeated to increase the second threshold value of a predetermined layer; as the second control, a calculation process is performed based on the increased second threshold, a first determination is performed to determine whether the metadata in the calculation result has changed with respect to the first control, and when it is determined in the first determination that there has been no change, the second threshold is repeatedly increased; As the second control, the recognition accuracy obtained by the calculation process is replaced with a higher recognition accuracy stored in the memory; instructing to decrease the second threshold value of a predetermined layer when it is determined that the second threshold value has changed in the first determination; performing a calculation process based on the reduced second threshold, performing a second determination to determine a change in the metadata in the calculation result relative to the first control, and when it is determined in the second determination that there is no change, setting the second threshold as a steady threshold, and when it is determined that there is a change, returning to the first control; A data processing method that causes a computer to execute processing.

6. A method for setting a decimal point position of N-bit (N is a natural number of 2 or more) fixed-length data corresponding to each of a plurality of layers constituting a multilayer neural network, the method including: setting the decimal point position of the fixed-length data; the multi-layer neural network is configured to perform arithmetic processing corresponding to each of the plurality of layers constituting the multi-layer neural network on each of the continuously input processing target data in accordance with a processing algorithm of the multi-layer neural network for the N-bit fixed-length data in which the decimal point position is set, the arithmetic processing determining unit 104 determines equivalent data which increases according to a saturation rate of the processing target data at the time of processing in the arithmetic processing, skips the second and subsequent arithmetic processing of the equivalent data, and detects metadata including the number of objects detected in the processing target data at the time of processing in the arithmetic processing, frames surrounding the objects, attributes, and recognition accuracy of the attributes; In the process of the calculation process for each layer, an upper limit counter value is counted as the number of times an upper limit value of a range determined by the decimal point position is exceeded, and a lower limit counter value is counted as the number of times a lower limit value of the range is exceeded, as a first control, a decimal point position is determined for each layer so that the upper limit counter value falls within a range of a first threshold value and the lower limit counter value falls within a range of a second threshold value, and an arithmetic process is performed on the processing target data at the processing time; As a second control to be repeated, an instruction is given to move the decimal point position of a predetermined layer by a bits (a>0) toward the least significant bit side; performing a calculation process using the decimal point position after the second control has been moved to the least significant bit side, performing a first determination to determine a change in the metadata in the calculation result relative to the first control, and repeating the movement to the least significant bit side when it is determined in the first determination that there has been no change; As the second control, the recognition accuracy obtained by the calculation process is replaced with a higher recognition accuracy stored in the memory; instructing to move the decimal point position of a predetermined layer by a bits (a>0) toward the most significant bit side when it is determined in the first determination that there is a change; A calculation process is performed using the decimal point position after the move to the most significant bit side, and a second determination is performed to determine a change in the metadata in the calculation result with respect to the first control. If it is determined in the second determination that there has been no change, the second threshold value is set as a steady threshold value, and if it is determined that there has been a change, the process returns to the first control. A data processing method that causes a computer to execute processing.

7. A data processing program for causing a computer to function as each part of the data processing device according to any one of claims 1 to 4.

Images