OBJECT DETECTION DEVICE, OBJECT DETECTION METHOD, AND OBJECT DETECTION PROGRAM
The object detection device optimizes decimal point positions in fixed-point neural networks by balancing saturated layers through dynamic threshold adjustment, enhancing detection accuracy and stability.
Patent Information
- Application Number
- JP2023523790
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-05-26
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2041-05-26
Smart Images

Figure 0007582464000001 
Figure 0007582464000002 
Figure 0007582464000003
Abstract
Description
[Technical field]
[0001] The disclosed technology relates to an object detection device, an object detection method, and an object detection program. [Background technology]
[0002] Various data processing technologies using multi-layer neural networks and deep learning have been developed. The range of applications is expanding to various fields such as recognition and detection. For example, object detection is a technology that detects metadata consisting of the position (square frame surrounding the object) and attributes (type of object such as person, car, etc.) of an object contained in an input image, and the detection accuracy of each object. In recent years, YOLO (You Only Look Once) (see Non-Patent Document 1) and SSD (Single Shot Multibox Detector) (see Non-Patent Document 2) that detect metadata based on the results of deep learning have been disclosed for object detection, and their application to object detection systems that require real-time performance such as autonomous driving and surveillance cameras is being considered.
[0003] From the viewpoint of network traffic load balancing and security protection, object detection devices for autonomous driving and surveillance cameras are being considered for installation at the edge of the network, i.e., on the terminal side. In order to install object detection devices at the edge of the network, it is essential that the devices are compact and consume less power.
[0004] In order to achieve miniaturization and low power consumption, a configuration has been proposed in which object detection processing based on deep learning is implemented in hardware and the data bit width of each data handled by a computing unit is reduced (see Non-Patent Document 3). The data includes input, output (feature map), weight (kernel), and bias. In general, when inference processing of deep learning is performed by software, each data used in product-sum calculation is treated as 32-bit floating-point data. This is because the range of values that each data can take is wide, and the range differs for each image or for each layer such as the convolution layer that constitutes the convolutional neural network. Non-Patent Document 3 reports that the effect of reducing the circuit size and power can be obtained by determining the data bit width in each layer of the convolutional neural network in advance using statistical information and reducing it to 8 to 16 bits.
[0005] In contrast to these approaches, a method has been disclosed in which each data width is uniformly set to a fixed-point number of n bits (n<32) and the decimal point position is dynamically controlled for each input image and for each layer (see Non-Patent Document 4). In the object detection device of Non-Patent Document 4, the decimal point position is dynamically changed for each input image and for each layer, thereby changing the value range according to the tendency of the calculation result for each layer and suppressing deterioration of recognition accuracy. In the conventional configuration, in addition to the object detection calculation unit that performs the inference process, it is equipped with a decimal point position control unit that determines the decimal point position of each layer, and upper limit / lower limit counters. The upper limit / lower limit counters are counters that measure the number of times the detection result exceeds the upper limit value (all bits all 1) of the value range (upper limit saturation) / the number of times it falls below the lower limit value (only the lowest bit is 1) for each layer during the calculation process in object detection. The upper limit / lower limit counter threshold (UP th / UN th ) is set, and the layer that exceeds the threshold is treated as the layer for which the decimal point position is to be changed, and the decimal point position is adjusted so that the counter value falls within the threshold.
[0006] <Non-Patent Document 1> Joseph Redmon et.al, “YOLOv3: An Incremental Improvement”, https: / / arxiv.org / abs / 1804.02767 <Non-Patent Document 2> Wei Liu et.al, “SSD:Single Shot MultiBox Detector”, https: / / arxiv.org / pdf / 1512.02325.pdf <Non-Patent Document 3> Zhisheng Li et.al, “Laius: An 8-Bit Fixed-Point CNN Hardware Inference Engine”2017 IEEE International Symposium on Parallel and Distributed Processing with Applications and 2017 IEEE International Conference on Ubiquitous Computing and Communications (ISPA / IUCC), Guangzhou, 2017,pp. 143-150,doi: 10.1109 / ISPA / IUCC.2017.00030. <Non-Patent Document 4> A. Hatta, H. Uzawa, S. Yoshida, T. Nitta, “Proposal of a dynamic decimal point position control method for object detection AI inference hardware,” Institute of Electronics, Information and Communication Engineers, September 2020. Summary of the Invention [Problem to be solved by the invention]
[0007] However, the quality of object detection results, including detection accuracy, is determined by the balance between the number of saturated layers, which is the number of layers where upper saturation occurs, and the average number of saturation times. Even if the average number of saturation times is reduced, the detection results will deteriorate if the number of saturated layers increases due to changing the decimal point position. In conventional methods, the threshold is the same for all input images, so there are cases where the number of saturated layers increases depending on the image, and the deterioration of detection results cannot be suppressed.
[0008] The disclosed technology has been developed in consideration of the above-mentioned points, and aims to provide an object detection device, an object detection method, and an object detection program that can suppress deterioration in accuracy of the detection results of object detection in an input image. [Means for solving the problem]
[0009] In order to achieve the above object, an object detection device according to one aspect of the present disclosure includes an object detection calculation unit that performs calculation processing corresponding to each of a plurality of layers constituting a multilayer neural network, to which an input image is input, on fixed-length data in which a decimal point position is set, in accordance with a processing algorithm of the multilayer neural network to which an input image is input; a saturation count counter that counts an upper limit saturation count, which is the number of times an upper limit value of a range determined by the decimal point position, and a lower limit saturation count, which is the number of times a value falls below a lower limit value of the range, in the calculation processing; an upper limit saturation layer count, which is the number of layers in which the upper limit saturation count is one or more, and a lower limit saturation count, which is the number of layers in which the upper limit saturation count is one or more. The system includes a saturated layer number counter that counts a lower saturated layer number, which is the number of layers whose saturation count has been one or more; a threshold determination unit that determines whether or not at least one of an upper saturation threshold, which is a threshold for the upper saturation count, and a lower saturation threshold, which is a threshold for the lower saturation count, is optimal based on the amount of change in the upper saturated layer number and the amount of change in the lower saturated layer number counted by the saturated layer number counter, and changes at least one of the upper saturation threshold and the lower saturation threshold when it is determined that the optimal value is not optimal; and a decimal point position control unit that sets the decimal point position for each of the plurality of layers based on the determination result of the threshold determination unit.
[0010] Furthermore, in order to achieve the above-mentioned object, an object detection method according to one embodiment of the present disclosure includes a computer performing arithmetic processing corresponding to each of a plurality of layers constituting a multilayer neural network on fixed-length data in which a decimal point position is set, in accordance with a processing algorithm of the multilayer neural network to which an input image is input, and in the arithmetic processing, counting each of the upper limit saturation count, which is the number of times the upper limit of a value range determined by the decimal point position, and the lower limit saturation count, which is the number of times the upper limit saturation count is exceeded, and the lower limit saturation count, which is the number of times the lower limit of the value range determined by the decimal point position, and counting each of the upper limit saturation layer number, which is the number of layers in which the upper limit saturation count has occurred once or more, and the lower limit saturation layer number, which is the number of layers in which the lower limit saturation count has occurred once or more, and based on the amount of change in the counted upper saturation layer number and the amount of change in the counted lower limit saturation layer number, if at least one of the upper limit saturation threshold, which is a threshold for the upper limit saturation count, and the lower limit saturation threshold, which is a threshold for the lower limit saturation count, is not optimal, changing at least one of the upper limit saturation threshold and the lower limit saturation threshold, and performing a process of setting the decimal point position for each of the plurality of layers based on a determination result of whether at least one of the upper limit saturation threshold and the lower limit saturation threshold is optimal.
[0011] Furthermore, in order to achieve the above-mentioned object, an object detection program according to one embodiment of the present disclosure causes a computer to perform arithmetic processing corresponding to each of a plurality of layers constituting a multilayer neural network, on fixed-length data in which a decimal point position is set, in accordance with a processing algorithm of the multilayer neural network to which an input image is input, and in the arithmetic processing, count an upper limit saturation count, which is the number of times the upper limit of a value range determined by the decimal point position, and a lower limit saturation count, which is the number of times the upper limit saturation count exceeds a lower limit of the value range determined by the decimal point position, and count an upper limit saturated layer number, which is the number of layers in which the upper limit saturation count has been reached once or more, and a lower limit saturated layer number, which is the number of layers in which the lower limit saturation count has been reached once or more, and based on the amount of change in the counted upper limit saturation layer number and the amount of change in the counted lower limit saturation layer number, if at least one of an upper limit saturation threshold which is a threshold for the upper limit saturation count and a lower limit saturation threshold which is a threshold for the lower limit saturation count is not optimal, change at least one of the upper limit saturation threshold and the lower limit saturation threshold, and execute a process of setting the decimal point position for each of the plurality of layers based on a determination result of whether at least one of the upper limit saturation threshold and the lower limit saturation threshold is optimal. Effect of the Invention
[0012] According to the disclosed technology, it is possible to suppress deterioration in accuracy of the detection result of object detection in an input image. [Brief description of the drawings]
[0013] [Figure 1] 1 is a functional block diagram of an object detection device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a hardware configuration of the object detection device. [Diagram 3] 4 is a flowchart of an object detection process according to the first embodiment. [Figure 4] 11A and 11B are diagrams for explaining the upper limit saturated layer number before and after a conventional change in the decimal point position. [Diagram 5] 11A and 11B are diagrams for explaining the upper limit saturated layer number before and after changing the decimal point position in the disclosed technology. [Figure 6]FIG. 11 is a functional block diagram of an object detection device according to a second embodiment. [Figure 7] 10 is a flowchart of an object detection process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals.
[0015] First Embodiment
[0016] The object detection device according to the first embodiment is a device that detects metadata including the position, attributes, and detection accuracy of each object included in an input image. Here, the position of an object is represented by at least one of the coordinates of the center of the object in the input image and a rectangular frame (bounding box) surrounding the object, for example. The attribute of an object is the type of object, such as a person or a car, and is sometimes called a category. The detection accuracy of an object is, for example, the probability that a detected object has a specific attribute.
[0017] As shown in FIG. 1, an object detection device 10 according to the first embodiment includes an object detection calculation unit 12, a decimal point position control unit 14, a saturation number counter 16, a threshold determination unit 18, and a saturation layer number counter 20.
[0018] The object detection calculation unit 12 performs calculation processing on the input image based on inference processing of deep learning.
[0019] Specifically, the object detection calculation unit 12 is a calculation processing circuit configured to perform calculation processing corresponding to each of a plurality of layers constituting a multilayer neural network according to a processing algorithm of the multilayer neural network. The processing by the multilayer neural network in the object detection calculation unit 12 typically uses a convolutional neural network (CNN).
[0020] To give an overview of CNN, CNN includes a feature extraction part that creates a feature map by alternating convolution layers that perform a convolution process of convolving a specified filter with an input image and pooling layers that perform a pooling process of downsizing the result of the convolution process, and an identification part that is made up of multiple fully connected layers and identifies objects contained in the input image from the feature map.
[0021] In the feature extraction part, a calculation is performed to convolve a filter with the image. In the identification part, a multiply-and-accumulate operation is performed to multiply the value of each pixel in the feature map by a weight and take the sum, and a bias is added to the result of the multiply-and-accumulate operation, which is then input to an activation function to obtain an output. As the activation function, ReLU (Rectified Linear Unit) or the like is used. The weight and activation function parameter values can be determined by learning.
[0022] The object detection calculation unit 12 performs calculation processing corresponding to each of the multiple layers constituting the multilayer neural network on fixed-length data with a decimal point position set according to the processing algorithm of the multilayer neural network to which the input image is input. Specifically, the object detection calculation unit 12 performs convolution calculation and combination processing of inference processing using an object detection algorithm based on deep learning, such as YOLO (Non-Patent Document 1) or SSD (Non-Patent Document 2), and outputs metadata such as the position, attributes, and detection accuracy of an object included in the input image as a detection result. Since such an object detection calculation unit 12 performs many product-sum calculations, it is often realized using a multi-core CPU or a GPU (graphics processing unit). Note that the object detection calculation unit 12 may be realized, for example, by an FPGA (Field Programmable Gate Array).
[0023] In this embodiment, the data that the object detection calculation unit 12 handles in each layer of the multi-layer neural network, such as the input, output (feature map), bias, and weight, is fixed-length data having a bit width smaller than 32 bits, and has a data structure that allows each layer to have a different decimal point position.
[0024] More specifically, in this embodiment, the object detection calculation unit 12 performs calculation processing corresponding to each of the multiple layers constituting the multi-layer neural network on fixed-length data having a bit width of, for example, 8 bits. Note that the decimal point position of the data handled by the object detection calculation unit 12 is set for each layer by the decimal point position control unit 14 described later.
[0025] In the convolution calculation in the object detection calculation unit 12, for example, if the input is 8 bits and the weight is 8 bits, the result obtained by the product-sum calculation will be a maximum of 16 bits. The object detection calculation unit 12 adds a bias to the 16-bit calculation result, multiplies it by an activation function, and creates a 16-bit intermediate input map. Since the feature map will be the input to the next layer, the 16-bit intermediate input map is reduced to an 8-bit width and used as the feature map for the next layer input. Note that the number of layers, the activation function, and the method of adding the bias are appropriately selected for each object detection algorithm used, and do not limit the disclosed technology.
[0026] The decimal point position control unit 14 sets the position of the decimal point of fixed-length data that is the subject of calculation in the object detection calculation unit 12 (hereinafter simply referred to as the “decimal point position”).
[0027] Specifically, based on the output of object detection calculation unit 12, i.e., the result of object detection by the multilayer neural network, decimal point position control unit 14 sets the decimal point position of the fixed-length data to be calculated by object detection calculation unit 12, corresponding to each of the multiple layers constituting the multilayer neural network. The decimal point position set by decimal point position control unit 14 is notified to object detection calculation unit 12. Based on the notification from decimal point position control unit 14, object detection calculation unit 12 changes the decimal point position of the fixed-length data corresponding to each of the multiple layers constituting the multilayer neural network.
[0028] The decimal point position control unit 14 determines the decimal point position of each layer using the detection results output from the object detection calculation unit 12. For example, when detecting an object contained in an input image as a video, the objects contained in the continuously input input images change little by little, and it is rare for all of them to change in a short period of time. For this reason, the decimal point position control unit 14 does not set the decimal point position of each layer based on the detection result of only one input image, but instead optimizes the decimal point position of each layer little by little by repeating the process of calculating the decimal point position of each layer using the object detection results for multiple input images.
[0029] The saturation count counter 16 counts the upper limit saturation count, which is the number of times that the upper limit value (all bits all 1) of the range determined by the decimal point position set by the decimal point position control unit 14 is exceeded during the calculation process of the object detection calculation unit 12, for each layer.
[0030] In addition, the saturation count counter 16 counts, for each layer, the lower limit saturation count, which is the number of times that the calculation by the object detection calculation unit 12 falls below the lower limit (only the least significant bit is 1) of the range determined by the decimal point position set by the decimal point position control unit 14. In the following, when there is no distinction between the upper limit saturation count and the lower limit saturation count, they may be simply referred to as the saturation count.
[0031] The threshold determination unit 18 optimizes the upper saturation threshold and the lower saturation threshold of the saturation count counter 16. The decimal point position control unit 14, which will be described in detail later, sets the decimal point position for each of the multiple layers of the multilayer neural network based on the determination result of the threshold determination unit 18. In the following description, the upper saturation threshold and the lower saturation threshold may be simply referred to as thresholds when there is no need to distinguish between them.
[0032] The saturated layer number counter 20 counts the upper limit saturated layer number, which is the number of layers for which the upper limit saturation count counted by the saturation count counter 16 has become 1 or more. The saturated layer number counter 20 also counts the lower limit saturated layer number, which is the number of layers for which the lower limit saturation count counted by the saturation count counter 16 has become 1 or more. In the following description, when there is no need to distinguish between the upper limit saturated layer number and the lower limit saturated layer number, they may simply be referred to as the saturated layer number.
[0033] Fig. 2 is a block diagram showing a hardware configuration of the object detection device 10. As shown in Fig. 2, the object detection device 10 includes a computer 30. The computer 30 includes a central processing unit (CPU) 30A, a read only memory (ROM) 30B, a random access memory (RAM) 30C, and an input / output interface (I / O) 30D. The CPU 30A, the ROM 30B, the RAM 30C, and the I / O 30D are connected to each other via a system bus 30E. The system bus 30E includes a control bus, an address bus, and a data bus.
[0034] In addition, a communication unit 32 and a storage unit 34 are connected to the I / O 30D.
[0035] The communication unit 32 is an interface for performing data communication with an external device.
[0036] The storage unit 34 is configured with a non-volatile storage device such as a hard disk, and stores an object detection program 34A (described later) and the like. The CPU 30A loads the object detection program 34A stored in the storage unit 34 into the RAM 30C and executes it. The storage unit 34 may be, for example, a portable storage device that is detachable from the computer 30.
[0037] Next, the object detection process executed by the object detection device 10 will be described with reference to the flowchart shown in FIG.
[0038] In step S100, the threshold determination unit 18 initializes the decimal point position of each layer, and initializes the upper saturation threshold, which is the threshold of the upper saturation count, to a predetermined initial upper saturation threshold, and initializes the lower saturation threshold, which is the threshold of the lower saturation count, to a predetermined initial lower saturation threshold, in order to obtain the change amount of the upper saturation layer count and the change amount of the lower saturation layer count before and after changing the decimal point position, and executes object detection twice. Specifically, in the first object detection, the object detection calculation unit 12 calculates the above-mentioned metadata included in the input image and outputs it to the saturation count counter 16 and the decimal point position control unit 14. In addition, the saturation count counter 16 counts the upper saturation count and the lower saturation count for each layer, and outputs it to the saturated layer count counter 20. Note that the upper saturation threshold and the lower saturation threshold are integers equal to or greater than 0. In addition, the upper saturation threshold and the lower saturation threshold may be the same value or different values.
[0039] The saturated layer number counter 20 counts an upper saturated layer number, which is the number of layers whose upper saturated number of times is 1 or more, and a lower saturated layer number, which is the number of layers whose lower saturated number of times is 1 or more. count do.
[0040] In the second object detection, the decimal point position control unit 14 adjusts the number of layers in which the upper limit saturation count exceeds the upper saturation threshold based on the detection result of the first object detection so as to reduce the number of layers in which the upper limit saturation count exceeds the upper saturation threshold and underThe decimal point position is set for each layer so that the number of layers that exceed the upper saturation threshold is reduced. That is, for layers whose upper saturation count exceeds the upper saturation threshold, the decimal point position is set so that the upper limit of the value range of the layer is increased, and for layers whose lower saturation count exceeds the lower saturation threshold, the decimal point position is set so that the lower limit of the value range of the layer is decreased. That is, the decimal point position is set so that the value range is widened.
[0041] In step S102, the saturated layer number counter 20 calculates the amount of change in the upper saturated layer number and the amount of change in the lower saturated layer number based on the first object detection result and the second object detection result. Here, the amount of change in the upper saturated layer number is the amount of increase in the upper saturated layer number in the second object detection relative to the upper saturated layer number in the first object detection. Also, the amount of change in the lower saturated layer number is the amount of increase in the lower saturated layer number in the second object detection relative to the lower saturated layer number in the first object detection.
[0042] In step S104, threshold determination unit 18 determines whether the amount of change in the upper saturated layer number obtained in step S102 is within an allowable range, i.e., whether the upper saturation threshold is an optimal upper saturation threshold. Specifically, if the amount of change in the upper saturated layer number is equal to or less than a predetermined upper limit change threshold, it is determined that the upper saturation threshold is optimal. Here, the upper limit change threshold is an integer equal to or greater than 1. On the other hand, if the amount of change in the upper saturated layer number is greater than the upper limit change threshold, it is determined that the upper saturation threshold is not optimal.
[0043] The threshold determination unit 18 also determines whether the change in the lower limit saturation layer number obtained in step S102 is within an allowable range, i.e., whether the lower limit saturation threshold is an optimal lower limit saturation threshold. Specifically, if the change in the lower limit saturation layer number is equal to or less than a predetermined lower limit change threshold, the lower limit saturation threshold is determined to be optimal. Here, the lower limit change threshold is an integer equal to or greater than 1. On the other hand, if the change in the lower limit saturation layer number is greater than the lower limit change threshold, the lower limit saturation threshold is determined to be not optimal. The upper limit change threshold and the lower limit change threshold are appropriately set according to the network model or application to which the object detection device 10 is applied. In this way, the threshold determination unit 18 determines whether the upper limit saturation threshold and the lower limit saturation threshold are optimal using the change in the upper limit saturation layer number and the change in the lower limit saturation layer number in two object detections by the object detection calculation unit 12.
[0044] If it is determined that at least one of the upper saturation threshold and the lower saturation threshold is not optimal, the process proceeds to step S105, and if it is determined that both the upper saturation threshold and the lower saturation threshold are optimal, the process proceeds to step S114.
[0045] In step S105, the threshold determination unit 18 changes at least one of the upper saturation threshold and the lower saturation threshold determined to be not optimal by increasing it by a predetermined increase value, and outputs it to the saturation count counter 16. Here, the increase value is an integer equal to or greater than 1. In this way, when it is determined that at least one of the upper saturation threshold and the lower saturation threshold is not optimal, the threshold determination unit 18 increases at least one of the upper saturation threshold and the lower saturation threshold determined to be not optimal. Note that the increase value by which at least one of the upper saturation threshold and the lower saturation threshold determined to be not optimal is increased may be set for each of the multiple layers, rather than being set to the same value for all of the multiple layers.
[0046] Steps S108 and S110 are similar to steps S100 and S102, but differ in that at least one of the upper saturation threshold and the lower saturation threshold changed in step S105 is used.
[0047] In step S112, the saturated layer number counter 20 judges the degree of change of the input image. In this embodiment, the degree of change of the input image is judged using the change amount of the upper saturated layer number and the lower saturated layer number, utilizing the fact that at least one of the upper saturated layer number and the lower saturated layer number may be different even at the same decimal point position depending on the type of input image.
[0048] Specifically, it is determined whether the amount of change in the upper saturated layer number calculated in step S110 is greater than a predetermined upper limit change threshold, and whether the amount of change in the lower saturated layer number calculated in step S110 is greater than a predetermined lower limit change threshold. The upper limit change threshold and the lower limit change threshold are integers equal to or greater than 1. If the amount of change in the upper saturated layer number calculated in step S110 is greater than the upper limit change threshold or if the amount of change in the lower saturated layer number calculated in step S110 is greater than the lower limit change threshold, it is determined that the degree of change in the input image is large, and the process proceeds to step S100. On the other hand, if the amount of change in the upper saturated layer number calculated in step S102 is equal to or less than the upper limit change threshold and the amount of change in the lower saturated layer number calculated in step S102 is equal to or less than the lower limit change threshold, it is determined that the degree of change in the input image is not large, that is, the degree of change in the input image is 0 or small, and the process proceeds to step S104. In this way, the saturated layer number counter 20 determines that the degree of change in the input image is large when at least one of the following cases is true: the amount of change in the upper saturated layer number between the previous object detection and the current object detection is greater than the upper limit change amount threshold, and the amount of change in the lower saturated layer number between the previous object detection and the current object detection is greater than the lower limit change amount threshold.
[0049] In this way, when at least one of the upper saturation threshold and the lower saturation threshold is not optimal and the degree of change in the input image is not large, the object detection process is repeated by changing at least one of the upper saturation threshold and the lower saturation threshold until the upper saturation threshold and the lower saturation threshold become optimal.
[0050] On the other hand, if it is determined in step S104 that the upper saturation threshold and the lower saturation threshold are optimal, then in step S114, object detection calculation unit 12 executes object detection n times. At this time, decimal point position control unit 14 changes the decimal point position every time object detection is performed. Note that n is an integer of 2 or more.
[0051] Specifically, object detection is performed by changing the decimal point position of each layer so that the average upper limit saturation count and the average lower limit saturation count of each layer are reduced. That is, for layers where the calculation result exceeds the upper limit of the value range of that layer, the decimal point position is moved in the direction where the calculation result is equal to or less than the upper limit, and for layers where the calculation result is below the lower limit, the decimal point position is moved in the direction where the calculation result is equal to or more than the lower limit. The decimal point positions obtained by performing object detection n times are stored.
[0052] In step S116, decimal point position control unit 14 determines the decimal point position when the detection result is the best, i.e., the decimal point position when object detection is performed with the fewest upper limit saturation counts and lower limit saturation counts, as the optimal decimal point position. In this way, when it is determined that the upper limit saturation threshold and the lower limit saturation threshold are optimal, decimal point position control unit 14 sets the decimal point position based on the upper limit saturation count and the lower limit saturation count in multiple object detections performed by object detection calculation unit 12.
[0053] In step S118, the object detection calculation unit 12 executes object detection using the decimal point position determined in step S116.
[0054] In step S120, it is determined whether or not there is a large degree of change between the previously input image and the currently input image. This determination is made in the same manner as in step S112.
[0055] If the degree of change in the input image is large, the process proceeds to step S100. On the other hand, if the degree of change in the input image is not large, the process proceeds to step S118 to repeat object detection. That is, object detection is repeated at the decimal point position determined in step S116 until the degree of change in the input image becomes large.
[0056] In this way, the upper saturation threshold and the lower saturation threshold are optimized in the processes of steps S104 to S112, and then the optimal decimal point position is determined in the processes of steps S114 and S116. This makes it possible to adjust the balance between the upper saturation count and the lower saturation count according to the input image, and the upper saturation layer count and the lower saturation layer count. This makes it possible to determine the optimal decimal point position for each of various input images, and makes it possible to suppress deterioration in the accuracy of the object detection results.
[0057] Here, for example, as shown in the upper graph of Fig. 4, when the number of layers in a multilayer neural network is six and the number of upper saturation layers is three, if the decimal point position is changed for three layers, layer numbers 1, 3, and 5, in which the upper saturation count exceeds the upper saturation threshold, as in the technology described in Non-Patent Document 4, the upper saturation count becomes equal to or less than the upper saturation threshold in all six layers, but the number of upper saturation layers increases to six, as shown in the lower graph of Fig. 4. For this reason, it is not possible to suppress deterioration in the accuracy of the detection result of object detection in the input image.
[0058] In contrast, according to the disclosed technology, instead of fixing the upper saturation threshold, if the upper saturation threshold is not optimal, the upper saturation threshold is increased, as shown in Fig. 5. Therefore, as shown in the upper graph in Fig. 5, the upper saturation count exceeds the upper saturation threshold only in one layer with layer number 3, and the decimal point position is changed only in this layer. As a result, as shown in the lower graph in Fig. 5, the number of upper saturation layers becomes four, and compared to the case in Fig. 4, it is possible to suppress deterioration in the accuracy of the detection result of object detection in the input image.
[0059] <Second embodiment>
[0060] Next, a second embodiment of the disclosed technique will be described. Note that the same parts as those in the first embodiment are given the same reference numerals, and detailed description will be omitted.
[0061] In the first embodiment, an example has been shown in which degradation of detection results is suppressed for any input image by using a configuration and method for determining the optimal value of the upper threshold before changing the decimal point position so that it does not exceed the upper saturation threshold and the lower saturation threshold set for each of the multiple layers of a multilayer neural network.
[0062] However, in the first embodiment, at least one of the upper saturation threshold and the lower saturation threshold is simply increased in step S105 in Fig. 3. Therefore, the increased threshold is not the optimal threshold, that is, in the case of an input image for which the optimal threshold cannot be determined, the optimal decimal point position cannot be determined unless the type of input image is changed. In this way, if the threshold continues to increase, the state in which the optimal decimal point position cannot be determined will continue.
[0063] Therefore, in this embodiment, a method is described that can determine the optimal decimal point position even for input images where the optimal threshold cannot be determined by simply increasing the threshold, by adding a process to decrease the threshold when the optimal threshold cannot be determined.
[0064] 6 shows the configuration of an object detection device 11 according to this embodiment. The object detection device 11 has a configuration in which a change parameter storage unit 22 is added to the object detection device 10 described in the first embodiment.
[0065] The change parameter storage unit 22 stores change parameters, which are the number of changes to at least one of the upper saturation threshold and the lower saturation threshold when the threshold is determined to be not optimal, and at least one of the amount of change in the upper saturation layer number and the amount of change in the lower saturation layer number when the threshold is determined to be not optimal.
[0066] FIG. 7 shows a flowchart of the object detection process according to this embodiment.
[0067] The object detection process in FIG. 7 is a process in which steps S106, S107, and S113 are added to the object detection process in FIG.
[0068] In step S106, the number of times at least one of the upper saturation threshold and the lower saturation threshold changed in step S105 and at least one of the amount of change in the upper saturated layer number and the amount of change in the lower saturated layer number determined in step S104 are stored in the change parameter memory unit 22.
[0069] In step S107, it is determined whether the number of changes to at least one of the upper saturation threshold and the lower saturation threshold stored in step S106 has reached the maximum number of changes Nmax, where Nmax is an integer equal to or greater than 1. If the number of changes to at least one of the upper saturation threshold and the lower saturation threshold has reached the maximum number of changes Nmax, the process proceeds to step S113, and if neither the number of changes to the upper saturation threshold nor the lower saturation threshold has reached the maximum number of changes Nmax, the process proceeds to step S108.
[0070] In step S113, each time the threshold is changed, at least one of the upper saturation threshold and the lower saturation threshold is set corresponding to the smallest amount of change in the upper saturation layer number or the smallest amount of change in the lower saturation layer number or ...
[0071] In this way, if the threshold is not optimal even when the number of changes reaches the maximum number of changes Nmax, at least one of the upper saturation threshold and the lower saturation threshold is set corresponding to at least one of the amount of change in the upper saturation layer number and the amount of change in the lower saturation layer number that is the smallest among at least one of the amount of change in the upper saturation layer number and the amount of change in the lower saturation layer number stored each time the threshold is increased. This makes it possible to determine the optimal decimal point position even for an input image for which the optimal threshold cannot be determined simply by increasing the threshold.
[0072] In the above embodiment, the object detection program is stored (installed) in advance in the storage device, but the present invention is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.
[0074] The following supplementary notes are further disclosed regarding the above embodiment.
[0075] (Additional note 1) Memory, at least one processor coupled to the memory; Including, The processor, Enter performing arithmetic processing corresponding to each of a plurality of layers constituting the multilayer neural network on fixed-length data in which a decimal point position is set, according to a processing algorithm of the multilayer neural network to which the force image is input; In the calculation process, an upper limit saturation count is counted as the number of times the value exceeds an upper limit value of a range determined by the decimal point position, and a lower limit saturation count is counted as the number of times the value falls below a lower limit value of the range, Counting an upper limit saturated layer number, which is the number of layers for which the upper limit saturated number of times is 1 or more, and a lower limit saturated layer number, which is the number of layers for which the lower limit saturated number of times is 1 or more, changing at least one of an upper saturation threshold, which is a threshold for the upper saturation count, and a lower saturation threshold, which is a threshold for the lower saturation count, when the upper saturation threshold and the lower saturation threshold are not optimal based on the amount of change in the upper saturation layer number and the amount of change in the lower saturation layer number; The decimal point position is set for each of the plurality of layers based on a determination result as to whether or not at least one of the upper saturation threshold and the lower saturation threshold is optimal. It is configured as follows: Object detection Device.
[0076] (Additional note 2) A non-transitory storage medium storing a program executable by a computer to perform an object detection process, The object detection process includes: The computer performing arithmetic processing corresponding to each of a plurality of layers constituting the multilayer neural network on fixed-length data in which a decimal point position is set, according to a processing algorithm of the multilayer neural network to which an input image is input; In the calculation process, an upper limit saturation count is counted as the number of times the value exceeds an upper limit value of a range determined by the decimal point position, and a lower limit saturation count is counted as the number of times the value falls below a lower limit value of the range, Counting an upper limit saturated layer number, which is the number of layers for which the upper limit saturated number of times is 1 or more, and a lower limit saturated layer number, which is the number of layers for which the lower limit saturated number of times is 1 or more, changing at least one of an upper saturation threshold, which is a threshold for the upper saturation count, and a lower saturation threshold, which is a threshold for the lower saturation count, when the upper saturation threshold and the lower saturation threshold are not optimal based on the amount of change in the upper saturation layer number and the amount of change in the lower saturation layer number; The decimal point position is set for each of the plurality of layers based on a determination result as to whether or not at least one of the upper saturation threshold and the lower saturation threshold is optimal. Non-transitory storage medium. [Explanation of symbols]
[0077] 10, 11 Object detection device 12 Object detection calculation unit 14 Decimal point position control section 16 Saturation counter 18 Threshold judgment unit 20 Saturation Layer Counter 22 Changed parameter storage section 30 Computers 34A Object Detection Program
Claims
1. A memory; at least one processor coupled to the memory; Including, The processor, a first process of performing object detection to detect an object included in an input image by performing arithmetic processing corresponding to each of a plurality of layers constituting a multilayer neural network on fixed-length data having a set decimal point position in accordance with a processing algorithm of the multilayer neural network to which an input image is input; a second process of counting an upper limit saturation count, which is the number of times that the calculation result of the calculation process exceeds an upper limit value of a range determined by the decimal point position, and a lower limit saturation count, which is the number of times that the calculation result of the calculation process falls below a lower limit value of the range, for each of the multiple layers; a third process of counting an upper limit saturated layer number, which is the number of layers whose upper limit saturation count is 1 or more, and a lower limit saturated layer number, which is the number of layers whose lower limit saturation count is 1 or more; a fourth process of setting a decimal point position so that the upper limit value of the range becomes larger for a layer in which the upper limit saturation count exceeds an upper limit saturation threshold, and setting a decimal point position so that the lower limit value of the range becomes smaller for a layer in which the lower limit saturation count exceeds a lower limit saturation threshold; a fifth process of performing the object detection a second time, and calculating an amount of change in the upper saturated layer number, which is an increase in the upper saturated layer number in the second object detection relative to the upper saturated layer number in the first object detection, and an amount of change in the lower saturated layer number, which is an increase in the lower saturated layer number in the second object detection relative to the lower saturated layer number in the first object detection; a sixth process for determining that the upper saturation threshold is optimal when the amount of change in the upper saturation layer number is equal to or less than a predetermined upper change amount threshold, determining that the upper saturation threshold is not optimal when the amount of change in the upper saturation layer number is greater than the upper change amount threshold, determining that the lower saturation threshold is optimal when the amount of change in the lower saturation layer number is equal to or less than a predetermined lower change amount threshold, and determining that the lower saturation threshold is not optimal when the amount of change in the lower saturation layer number is greater than the lower change amount threshold; a seventh process for changing the upper saturation threshold by increasing the upper saturation threshold by a predetermined increase value if it is determined that the upper saturation threshold is not optimal, and for changing the lower saturation threshold by increasing the lower saturation threshold by a predetermined increase value if it is determined that the lower saturation threshold is not optimal; an eighth process of determining whether or not the upper saturation threshold and the lower saturation threshold are optimal after executing the seventh process, and repeating the seventh process until it is determined that the upper saturation threshold and the lower saturation threshold are optimal; a ninth process of performing the object detection a plurality of times when it is determined that the upper limit saturation threshold and the lower limit change amount threshold are optimal, and each time the object detection is performed, for a layer in which the calculation result exceeds an upper limit value of the range, moving a decimal point position in a direction in which the calculation result is equal to or less than the upper limit value, and for a layer in which the calculation result is below a lower limit value of the range, moving a decimal point position in a direction in which the calculation result is equal to or greater than the lower limit value, and determining, for each of the plurality of layers, the decimal point position when the object detection was performed when the upper limit saturation count and the lower limit saturation count were the smallest as the optimal decimal point position; a tenth process of detecting the object using the optimal decimal point position; An object detection device that performs the above.
2. The processor sets an increment value for each of the plurality of layers that increases at least one of the upper saturation threshold and the lower saturation threshold that is determined to be non-optimal. The object detection device according to claim 1 .
3. The processor determines that a degree of change in the input image is large when at least one of a case where an amount of change in the upper saturated layer number between a previous object detection and a current object detection is greater than the upper change amount threshold and a case where an amount of change in the lower saturated layer number between a previous object detection and a current object detection is greater than the lower change amount threshold, When it is determined that the degree of change in the input image is large, the processor repeats the first process to the ninth process until the decimal point position is determined as the optimal decimal point position. The object detection device according to claim 1 or 2.
4. The memory stores the number of times the saturation thresholds have been changed in the seventh process among the number of times the upper saturation threshold and the lower saturation threshold have been changed, and the amount of change in the number of saturation layers determined to be not optimal in the sixth process among the amount of change in the upper saturation layer number and the amount of change in the lower saturation layer number, When the number of times the saturation threshold has been changed reaches a predetermined maximum number of times, the processor changes the saturation threshold to the saturation threshold corresponding to the smallest amount of change in the number of saturated layers among the stored amounts of change in the number of saturated layers. The object detection device according to any one of claims 1 to 3.
5. The computer a first process of performing object detection to detect an object included in an input image by performing arithmetic processing corresponding to each of a plurality of layers constituting a multilayer neural network on fixed-length data having a set decimal point position in accordance with a processing algorithm of the multilayer neural network to which an input image is input; a second process of counting an upper limit saturation count, which is the number of times that the calculation result of the calculation process exceeds an upper limit value of a range determined by the decimal point position, and a lower limit saturation count, which is the number of times that the calculation result of the calculation process falls below a lower limit value of the range, for each of the multiple layers; a third process of counting an upper limit saturated layer number, which is the number of layers whose upper limit saturation count is 1 or more, and a lower limit saturated layer number, which is the number of layers whose lower limit saturation count is 1 or more; a fourth process of setting a decimal point position so that the upper limit value of the range becomes larger for a layer in which the upper limit saturation count exceeds an upper limit saturation threshold, and setting a decimal point position so that the lower limit value of the range becomes smaller for a layer in which the lower limit saturation count exceeds a lower limit saturation threshold; a fifth process of performing the object detection a second time, and calculating an amount of change in the upper saturated layer number, which is an increase in the upper saturated layer number in the second object detection relative to the upper saturated layer number in the first object detection, and an amount of change in the lower saturated layer number, which is an increase in the lower saturated layer number in the second object detection relative to the lower saturated layer number in the first object detection; a sixth process for determining that the upper saturation threshold is optimal when the amount of change in the upper saturation layer number is equal to or less than a predetermined upper change amount threshold, determining that the upper saturation threshold is not optimal when the amount of change in the upper saturation layer number is greater than the upper change amount threshold, determining that the lower saturation threshold is optimal when the amount of change in the lower saturation layer number is equal to or less than a predetermined lower change amount threshold, and determining that the lower saturation threshold is not optimal when the amount of change in the lower saturation layer number is greater than the lower change amount threshold; a seventh process for changing the upper saturation threshold by increasing the upper saturation threshold by a predetermined increase value if it is determined that the upper saturation threshold is not optimal, and for changing the lower saturation threshold by increasing the lower saturation threshold by a predetermined increase value if it is determined that the upper saturation threshold is not optimal; an eighth process of determining whether or not the upper saturation threshold and the lower saturation threshold are optimal after executing the seventh process, and repeating the seventh process until it is determined that the upper saturation threshold and the lower saturation threshold are optimal; a ninth process of performing the object detection a plurality of times when it is determined that the upper limit saturation threshold and the lower limit change amount threshold are optimal, and each time the object detection is performed, for a layer in which the calculation result exceeds an upper limit value of the range, moving a decimal point position in a direction in which the calculation result is equal to or less than the upper limit value, and for a layer in which the calculation result is below a lower limit value of the range, moving a decimal point position in a direction in which the calculation result is equal to or greater than the lower limit value, and determining, for each of the plurality of layers, the decimal point position when the object detection was performed when the upper limit saturation count and the lower limit saturation count were the smallest as the optimal decimal point position; A tenth process of performing the object detection using the optimal decimal point position; An object detection method that performs
6. On the computer, a first process of performing object detection to detect an object included in an input image by performing arithmetic processing corresponding to each of a plurality of layers constituting a multilayer neural network on fixed-length data having a set decimal point position in accordance with a processing algorithm of the multilayer neural network to which an input image is input; a second process of counting an upper limit saturation count, which is the number of times that the calculation result of the calculation process exceeds an upper limit value of a range determined by the decimal point position, and a lower limit saturation count, which is the number of times that the calculation result of the calculation process falls below a lower limit value of the range, for each of the multiple layers; a third process of counting an upper limit saturated layer number, which is the number of layers whose upper limit saturation count is 1 or more, and a lower limit saturated layer number, which is the number of layers whose lower limit saturation count is 1 or more; a fourth process of setting a decimal point position so that the upper limit value of the range becomes larger for a layer in which the upper limit saturation count exceeds an upper limit saturation threshold, and setting a decimal point position so that the lower limit value of the range becomes smaller for a layer in which the lower limit saturation count exceeds a lower limit saturation threshold; a fifth process of performing the object detection a second time, and calculating an amount of change in the upper saturated layer number, which is an increase in the upper saturated layer number in the second object detection relative to the upper saturated layer number in the first object detection, and an amount of change in the lower saturated layer number, which is an increase in the lower saturated layer number in the second object detection relative to the lower saturated layer number in the first object detection; a sixth process for determining that the upper saturation threshold is optimal when the amount of change in the upper saturation layer number is equal to or less than a predetermined upper change amount threshold, determining that the upper saturation threshold is not optimal when the amount of change in the upper saturation layer number is greater than the upper change amount threshold, determining that the lower saturation threshold is optimal when the amount of change in the lower saturation layer number is equal to or less than a predetermined lower change amount threshold, and determining that the lower saturation threshold is not optimal when the amount of change in the lower saturation layer number is greater than the lower change amount threshold; a seventh process for changing the upper saturation threshold by increasing the upper saturation threshold by a predetermined increase value if it is determined that the upper saturation threshold is not optimal, and for changing the lower saturation threshold by increasing the lower saturation threshold by a predetermined increase value if it is determined that the upper saturation threshold is not optimal; an eighth process of determining whether or not the upper saturation threshold and the lower saturation threshold are optimal after executing the seventh process, and repeating the seventh process until it is determined that the upper saturation threshold and the lower saturation threshold are optimal; a ninth process of performing the object detection a plurality of times when it is determined that the upper limit saturation threshold and the lower limit change amount threshold are optimal, and each time the object detection is performed, for a layer in which the calculation result exceeds an upper limit value of the range, moving a decimal point position in a direction in which the calculation result is equal to or less than the upper limit value, and for a layer in which the calculation result is below a lower limit value of the range, moving a decimal point position in a direction in which the calculation result is equal to or greater than the lower limit value, and determining, for each of the plurality of layers, the decimal point position when the object detection was performed when the upper limit saturation count and the lower limit saturation count were the smallest as the optimal decimal point position; A tenth process of performing the object detection using the optimal decimal point position; An object detection program that executes the following:
Citation Information
Patent Citations
Arithmetic processing device, information processing device, method, and program
JP2018124681A
