OBJECT DETECTION DEVICE, OBJECT DETECTION METHOD, AND OBJECT DETECTION PROGRAM

Abstract

Provided is an object detection device 10 comprising: a metadata acquisition unit 103 that acquires metadata including at least reliability and the position of an object within an image from a convolutional neural network that has received input of the image; a holding unit 104 that holds a group of feature map values that are results of output by the convolutional neural network; and a feature map value acquisition unit 105 that, only if a reliability acquired by reading from the holding unit 104 a feature map value relating to the reliability among the group of feature map values held in the holding unit 104 is larger than a preset threshold value, acquires the position of a corresponding object by reading from the holding unit 104 a feature map value relating to the position of the object.

Description

[Technical Field]

[0001] The disclosed technology relates to an object detection device, an object detection method, and an object detection program. [Background technology]

[0002] There is an object identification device that outputs a bounding box (BB) consisting of the position coordinates of the object contained in the input image, its class (type of object, such as person or car), and its reliability. In recent years, YOLO (You Only Look Once) and SSD (Single Shot multibox Detector) have been disclosed, which enable the output of the BB using a single convolutional neural network (CNN). The application of object identification device technology to edge devices or terminals, such as surveillance cameras and drone control, is being considered.

[0003] In object detection based on CNN such as YOLO, detection processing is performed in the final layer to obtain BB based on the feature map values ​​obtained by the CNN calculations up to that point. Figure 1 shows the processing flow of CNN including detection processing. In YOLO or SSD, a predetermined B is calculated for each unit called a grid, which is obtained by dividing an image of W pixels horizontally and H pixels vertically. num BB(B[0]~B[B num The feature map values ​​corresponding to the coordinates of the BB (tx, ty, tw, th) and the reliability of the object presence at the coordinates (object reliability) (p obj ), and the value corresponding to the confidence level for each class of the object (p[0] to p[C num -1], C num The detection process converts these feature map values ​​into BBs, removes BBs whose object reliability obtained as a result of the conversion is below a threshold, and removes duplicate BBs (Non-Maximum-Suppression: NMS).

[0004] Methods for performing real-time object detection based on CNN have been disclosed (Non-Patent Documents 1 and 2). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] H. Nakahara et al, "A Demonstration of FPGA-Based You Only Look Once Version2 (YOLOv2)," 2018 28th International Conference on Field Programmable Logic and Applications (FPL), 2018, pp. 457-4571. [Non-patent document 2] H.Uzawa et al, "High-definition object detection technology based on AI inference scheme and its implementation", IEICE Electronics Express, 2021, Volume 18, Issue 22, Pages 2021032 Summary of the Invention [Problem to be solved by the invention]

[0006] In the above method, the calculations (convolution calculations, etc.) within the CNN until the feature map output by the CNN (CNN output feature map) is obtained are accelerated using dedicated hardware. On the other hand, the detection process, which takes the CNN output feature map, which is the output result of the CNN, as input, is implemented in software and is not accelerated. In addition, since the CNN output feature map is stored in DRAM (Dynamic Random Access Memory), the detection process must be performed by reading the feature map from the DRAM.

[0007] The disclosed technology has been made in consideration of the above points, and aims to provide an object detection device, an object detection method, and an object detection program that speed up detection processing compared to existing technologies. [Means for solving the problem]

[0008] A first aspect of the present disclosure is an object detection device including: a metadata acquisition unit that acquires metadata including at least a position and reliability of an object included in an image from a convolutional neural network to which the image is input; a storage unit that stores a set of feature map values ​​that are output results of the convolutional neural network; and a feature map value acquisition unit that reads a feature map value related to the reliability from the storage unit among the set of feature map values ​​stored in the storage unit and obtains the position of the object only when the reliability obtained by reading the feature map value related to the reliability from the storage unit exceeds a predetermined threshold.

[0009] A second aspect of the present disclosure is an object detection method, in which a processor acquires metadata including at least the position and reliability of an object included in an image from a convolutional neural network to which the image is input, stores a set of feature map values ​​that are the output results of the convolutional neural network, and, only when a feature map value related to the reliability among the stored set of feature map values ​​is read and the reliability obtained exceeds a predetermined threshold, reads a corresponding feature map value related to the position of the object from the storage unit and executes a process to obtain the position of the object.

[0010] A third aspect of the present disclosure is an object detection program that causes a computer to acquire metadata including at least a position and reliability of an object included in an image from a convolutional neural network to which the image is input, store a set of feature map values ​​that are the output results of the convolutional neural network, and, only when a feature map value related to the reliability among the set of stored feature map values ​​is read and the reliability obtained exceeds a predetermined threshold, read a corresponding feature map value related to the position of the object from the storage unit to obtain the position of the object. [Effects of the Invention]

[0011] According to the disclosed technology, it is possible to provide an object detection device, an object detection method, and an object detection program that perform detection processing at a higher speed than existing technologies. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing a processing flow of CNN including detection processing. [Figure 2] 10 is a flowchart illustrating detection processing performed by an object detection device as a comparative example of the embodiment. [Figure 3] FIG. 3 is a diagram illustrating the detection processing shown in FIG. 2. [Figure 4] FIG. 2 is a block diagram showing a hardware configuration of the object detection device. [Figure 5] FIG. 2 is a block diagram illustrating an example of a functional configuration of an object detection device. [Figure 6] 10 is a flowchart showing the flow of an object detection process performed by the object detection device. [Figure 7] FIG. 7 is a diagram for explaining the detection processing shown in FIG. 6. [Figure 8] 10 is a graph comparing the number of times the feature map is read out between the method according to the embodiment and the method of the comparative example. [Figure 9] 10 is a flowchart showing the flow of an object detection process performed by the object detection device. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0014] First, a detection process performed by an object detection device as a comparative example of this embodiment will be described. Fig. 2 is a flowchart showing the detection process performed by an object detection device as a comparative example of this embodiment.

[0015] In the detection process of the comparative example, the object detection device first initializes a variable n used in the detection process to n=0 (step S11). After initializing the variable n to n=0, the object detection device then num It is determined whether n is less than B (step S12). num If it is less than (step S12; Yes), the object detection apparatus then converts all feature map values ​​of B[n] into BB information (step S13).

[0016] FIG. 3 is a diagram for explaining the detection process shown in FIG. 2, and is a diagram for explaining step S13 of the flowchart shown in FIG. 2. The object detection device converts all feature map values ​​of B[n] into BB information in step S13 of FIG. 2, but the value corresponding to the object reliability (p obj ) and converts all channels into BB information. Ch=B num ×(5+C num ) The number 5 in the above formula is tx, ty, tw, th, p obj This corresponds to 5 channels.

[0017] After converting the feature map values ​​of B[n] into BB information, the object detection device then removes BBs whose object reliability is equal to or less than a threshold (step S14), and increments the variable n by one (step S15).

[0018] As a result of the determination in step S12, n is B num If so (step S12; No), the object detection device then removes overlapping BBs using NMS (step S16). NMS is a process of removing those with low scores when predicted BBs overlap.

[0019] In this way, in the detection process of the comparative example, all feature map values ​​of all channels of the CNN output feature map are read and each is converted into BB information. For example, if the width (W) and height (H) of the feature map value are 72 and B num 3, C num is set to 80, 72 × 72 × 255 = 1,321,920 feature map values ​​are read from the DRAM. Thus, in the detection process of the comparative example, the number of feature map values ​​read is enormous, resulting in a long processing time.

[0020] In this embodiment, an object detection device that can reduce the processing time compared to the detection processing of the comparative example is shown.

[0021] FIG. 4 is a block diagram showing the hardware configuration of the object detection device 10. As shown in FIG.

[0022] 4, the object detection device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0023] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the program stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores an object detection program that detects an object included in an image.

[0024] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0025] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs.

[0026] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may also function as the input unit 15 by adopting a touch panel system.

[0027] The communication interface 17 is an interface for communicating with other devices. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0028] Next, the functional configuration of the object detection device 10 will be described.

[0029] FIG. 5 is a block diagram showing an example of the functional configuration of the object detection device 10.

[0030] 5, the object detection device 10 has, as functional components, an image acquisition unit 101, a recognition unit 102, a metadata acquisition unit 103, a storage unit 104, a feature map value acquisition unit 105, and an output unit 106. Each functional component is realized by the CPU 11 reading out an object detection program stored in the ROM 12 or storage 14, expanding the program in the RAM 13, and executing the program.

[0031] The image acquisition unit 101 acquires an image of an object detection target.

[0032] The recognition unit 102 performs image processing on the image acquired by the image acquisition unit 101 and recognizes objects contained in the image. The recognition unit 102 inputs the image acquired by the image acquisition unit 101 into a convolutional neural network (CNN). The CNN outputs metadata including at least the position of the object contained in the image and the reliability of the object. The metadata is temporarily stored in the storage unit 104 by the metadata acquisition unit 103 (described later). From the stored metadata, metadata that satisfies predetermined conditions is read by the feature map value acquisition unit 105.

[0033] The metadata acquisition unit 103 acquires metadata including at least the position and reliability of an object included in the input image from the CNN to which the image is input. The reliability may be composed of a class reliability group for each object class. The reliability may further be composed of an object reliability indicating the probability of the existence of the object.

[0034] The storage unit 104 stores a group of feature map values ​​that are the output results of the CNN. The group of feature map values ​​is stored in a predetermined B num BB(B[0]~B[B num The storage unit 104 may be provided in the RAM 13, for example.

[0035] Only when the reliability obtained by reading a feature map value related to reliability from storage unit 104 among the group of feature map values ​​stored in storage unit 104 exceeds a predetermined threshold, feature map value acquisition unit 105 reads a feature map value related to the position of the corresponding object from storage unit 104 to obtain the position of the object. The threshold can be changed depending on the required detection accuracy.

[0036] The feature map value acquisition unit 105 reads out the feature map value related to the position of the corresponding object and the feature map value related to the class-specific reliability from the storage unit 104 only when the object reliability obtained from the feature map value related to the object reliability exceeds a threshold value.

[0037] The output unit 106 outputs the object recognition result obtained by the recognition unit 102. The image recognition result obtained by the recognition unit 102 may be output in a state in which it is superimposed on the input image. For example, as shown in FIG. 1, the output unit 106 may output the image recognition result in a state in which it is superimposed on an area corresponding to the object in the input image and the name of the detected object is superimposed in the frame.

[0038] Next, the operation of the object detection device 10 will be described.

[0039] 6 is a flowchart showing the flow of the object detection process by the object detection device 10. The object detection process is performed by the CPU 11 reading out the object detection program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it.

[0040] The flowchart shown in Fig. 6 shows detection processing for a CNN output feature map output by the CNN and stored in, for example, the RAM 13. Fig. 7 is a diagram for explaining the detection processing shown in Fig. 6.

[0041] The CPU 11 initializes a variable n used in the detection process to 0 (step S101).

[0042] Next, the CPU 11 checks whether the variable n is B numIt is determined whether the variable n is less than B (step S102). num If it is less than (step S102; Yes), the CPU 11 obj All feature map values ​​in the channel are converted into BB information (object reliability) (step S103).

[0043] Following step S103, the CPU 11 extracts grids whose object reliability is equal to or greater than a predetermined threshold (step S104).

[0044] Following step S104, the CPU 11 calculates p obj Channels other than tx, ty, tw, th, p[0]~p[C num The feature map values ​​of each channel (p[0] to p[C num -1] is a value corresponding to the reliability of each object class, and p[0]~p[C num −1] are collectively referred to as the class-specific confidence group.

[0045] Following step S105, the CPU 11 increments the variable n by one (step S106), and returns to the determination process of step S102.

[0046] As a result of the determination in step S102, n is B num If it is equal to or greater than this (step S102; No), the CPU 11 removes BBs whose object reliability obtained as a result of conversion to BB information is equal to or less than a threshold, and removes overlapping BBs (step S107). The CPU 11 removes BBs by NMS (Non-Maximum-Suppression). NMS is a process of removing those with low scores when predicted BBs overlap.

[0047] In this manner, in this embodiment, the object detection device 10 obj The feature map values ​​of the channel are read comprehensively, but the feature map values ​​of other channels are readobj The object is retrieved only if the object confidence obtained from exceeds a threshold.

[0048] By the series of processes shown in Fig. 6, the feature map value corresponding to the BB whose object reliability is below the threshold and is to be removed is read out as follows: obj In Fig. 7, the reading of the feature map value corresponding to BB, which has an object reliability below the threshold and is to be removed, is omitted, and the number of times of reading of the feature map value can be reduced. obj The number of BBs whose object reliability exceeds the threshold is K, and the number of classes is C. num Then, the number R of read feature map values ​​in this embodiment can be expressed by the following formula. R=K×(4+C num ) + W × H × B num

[0049] In the above formula, W×H×B num All p obj The size of the feature map in the channel is W × H, and p obj The number of B channels is the same as the number of BBs divided by the number of grids. num The number of channels per BB is p obj Excluding the channel, 4+C num Here, 4 corresponds to the four channels tx, ty, tw, and th. These channels are obj Since the object is read only if the object reliability obtained from exceeds the threshold, the number of reads is K × (4 + C num )

[0050] FIG. 8 is a graph comparing the number of times the feature map is read between the method according to this embodiment and the method according to the comparative example. In the method according to the comparative example, the number of times the feature map is read is constant because the feature map values ​​of all channels of all grids are read. In contrast, in the method according to this embodiment, the number of times is proportional to K. For example, when K=100, C num =80, B num= 3 and W = H = 72, then R = 23952. In this case, the number of reads in the method according to this embodiment is 1 / 50 or less of the 1,321,920 reads in the comparative example.

[0051] Depending on the type of CNN used by the object detection device 10 for object detection, the BB may not include object confidence. Even in such a case, a method for reducing the number of times the feature map is read out will be described below. Specifically, when the BB does not include object confidence, the object detection device 10 reads out the class confidence sets p[0] to p[C num The object detection device 10 comprehensively reads out the feature map values ​​of the grids corresponding to the other channels tx, ty, tw, and th from the per-class confidence groups p[0] to p[C num -1] is read out only when any of the class reliability values ​​(class reliability values) obtained from the above is equal to or greater than a threshold value.

[0052] 9 is a flowchart showing the flow of the object detection process by the object detection device 10. The object detection process is performed by the CPU 11 reading out the object detection program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it.

[0053] The flowchart shown in FIG. 9 shows detection processing for a CNN output feature map output by the CNN and stored in the RAM 13, for example.

[0054] The CPU 11 initializes a variable n used in the detection process to 0 (step S111).

[0055] Next, the CPU 11 checks whether the variable n is B num It is determined whether the variable n is less than B (step S112). num If it is less than this (step S112; Yes), the CPU 11 initializes a variable m used in the detection process to 0 (step S113).

[0056] Next, the CPU 11 checks whether the variable m is Cnum It is determined whether the variable m is less than C (step S114). num If it is less than (step S114; Yes), the CPU 11 converts all feature map values ​​in the p[m] channel in B[n] into BB information (class-specific reliability) (step S115).

[0057] Next, the CPU 11 increments the variable m by one (step S116), and returns to the determination in step S114.

[0058] As a result of the determination in step S114, the variable m is C num If it is equal to or greater than this (step S114; No), the CPU 11 then calculates the class confidence groups p[0] to p[C num −1] is equal to or greater than a threshold value (step S117).

[0059] Next, the CPU 11 extracts the class confidence scores p[0] to p[C num -1] The feature map values ​​of the channels other than the channel (tx, ty, tw, th) are read out and converted into BB information (step S118).

[0060] Following step S118, the CPU 11 increments the variable n by one (step S119), and returns to the determination process of step S112.

[0061] As a result of the determination in step S112, the variable n is B num If it is equal to or greater than this (step S112; No), the CPU 11 removes BBs whose object reliability obtained as a result of conversion to BB information is equal to or less than a threshold, and removes overlapping BBs (step S120). The CPU 11 removes BBs by NMS (Non-Maximum-Suppression). NMS is a process of removing those with low scores when predicted BBs overlap.

[0062] This series of processes omits the reading of feature map values ​​corresponding to BBs that are to be removed when the class confidence group is below the threshold, thereby reducing the number of times that feature map values ​​are read.

[0063] In the above embodiments, the object detection process executed by the CPU after reading the software (program) may be executed by various processors other than the CPU. Examples of such processors include programmable logic devices (PLDs) such as field-programmable gate arrays (FPGAs), whose circuit configuration can be changed after fabrication, and dedicated electrical circuits such as application-specific integrated circuits (ASICs), which are processors with circuit configurations specifically designed to execute specific processes. The object detection process may be executed by one of these processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0064] In addition, in each of the above embodiments, the object detection processing program is described as being pre-stored (installed) in the storage 14, but the present invention is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0065] The following additional notes are provided regarding the above-described embodiments. (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: acquiring metadata including at least the position and reliability of an object included in the image from a convolutional neural network to which the image is input; A set of feature map values ​​that are output results of the convolutional neural network is retained; only when the reliability obtained by reading out from the storage unit a feature map value relating to the reliability among the group of stored feature map values ​​exceeds a predetermined threshold, a process is executed in which a corresponding feature map value relating to the position of the object is read out from the storage unit to obtain the position of the object. The object detection device is configured as follows.

[0066] (Additional note 2) A non-transitory storage medium storing a program executable by a computer to perform an object detection process, The object detection process includes: acquiring metadata including at least the position and reliability of an object included in the image from a convolutional neural network to which the image is input; A set of feature map values ​​that are output results of the convolutional neural network is retained; only when the reliability obtained by reading out from the storage unit a feature map value relating to the reliability among the group of stored feature map values ​​exceeds a predetermined threshold, a process is executed in which a corresponding feature map value relating to the position of the object is read out from the storage unit to obtain the position of the object. Non-transitory storage medium. [Explanation of symbols]

[0067] 10. Object detection device 101 Image acquisition unit 102 Recognition part 103 Metadata Acquisition Unit 104 Holding part 105 Feature map value acquisition unit 106 Output section

Claims

1. a metadata acquisition unit that acquires metadata including at least the position and reliability of an object included in an image from a convolutional neural network to which the image is input; a storage unit that stores a group of feature map values ​​that are output results of the convolutional neural network; a feature map value acquisition unit that reads out from the storage unit a feature map value relating to the reliability of the group of feature map values ​​stored in the storage unit and acquires the position of the object only when the reliability obtained by reading out from the storage unit a feature map value relating to the corresponding reliability exceeds a predetermined threshold; Equipped with the reliability includes an object reliability indicating the certainty of the existence of the object, and a class reliability group for each class of the object; the feature map value acquisition unit reads out the feature map value related to the position of the corresponding object and the feature map value related to the class-specific reliability group from the storage unit only when the object reliability obtained from the feature map value related to the object reliability exceeds the threshold.

2. 2. The object detection device according to claim 1, wherein the feature map value acquisition unit reads out a feature map value relating to the position of the corresponding object from the storage unit only when at least one of the per-class confidence groups obtained from the feature map values ​​relating to the per-class confidence groups of the object exceeds the threshold.

3. The object detection device according to claim 1 or 2, further comprising an output unit that outputs a result of the object recognition by the convolutional neural network.

4. The processor: acquiring metadata including at least the position and reliability of an object included in the image from a convolutional neural network to which the image is input; A set of feature map values ​​that are output results of the convolutional neural network is retained; only when the reliability obtained by reading out a feature map value relating to the reliability from the group of stored feature map values ​​exceeds a predetermined threshold, a feature map value relating to the position of the corresponding object is read out to obtain the position of the object; the reliability includes an object reliability indicating the certainty of the existence of the object, and a class reliability group for each class of the object; Only when the object reliability obtained from the feature map value related to the object reliability exceeds the threshold, the feature map value related to the corresponding object position and the feature map value related to the class-specific reliability group are read out. The object detection method to perform the processing.

5. On the computer, acquiring metadata including at least the position and reliability of an object included in the image from a convolutional neural network to which the image is input; A set of feature map values ​​that are output results of the convolutional neural network is retained; only when the reliability obtained by reading out a feature map value relating to the reliability from the group of stored feature map values ​​exceeds a predetermined threshold, a feature map value relating to the position of the corresponding object is read out to obtain the position of the object; the reliability includes an object reliability indicating the certainty of the existence of the object, and a class reliability group for each class of the object; Only when the object reliability obtained from the feature map value related to the object reliability exceeds the threshold, the feature map value related to the corresponding object position and the feature map value related to the class-specific reliability group are read out. The object detection program that performs the processing.

Images