OBJECT DETECTION DEVICE, OBJECT DETECTION METHOD, AND OBJECT DETECTION PROGRAM

Abstract

The first detection unit (121) performs object detection on a target image (191) and calculates a detection result (192). The processing unit (130) fills in the target bounding box of the target image for each target bounding box to obtain a filled-in image group (193). The second detection unit (122) performs the object detection for each filled-in image to obtain a detection result group (194). The first removal unit (141) performs a first removal process on the detection result set to obtain a first result set (195). The second removal unit (142) performs a second removal process on the first result set to obtain a second result set (196) as an object detection result (197).

Description

[Technical field]

[0001] The present disclosure relates to a technology for detecting an object captured in an image. [Background technology]

[0002] The task of object detection involves indicating the location of each object in an input image with a bounding box and indicating the type of each object with a label. In recent years, deep learning methods using neural networks have achieved extremely high accuracy in object detection tasks.

[0003] Deep learning is used to build an image classifier (e.g., a multi-class classifier). Non-Patent Document 1 describes an adversarial examples attack that causes erroneous classification results by a multi-class classifier by adding perturbations to input data. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow and Rob Fergus: Intriguing properties of neural networks, in International Conference on Learning Representations (ICLR) (2014) Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure aims to reduce the effects of adversarial sample attacks. [Means for solving the problem]

[0006] The object detection device of the present disclosure is a first detection unit that performs object detection on a target image and calculates a detection result indicating a bounding box group; a processing unit that obtains a filled-in image group by filling in the target bounding box of the target image for each target bounding box selected from the bounding box group of the target image to generate a filled-in image; a second detection unit that performs the object detection on the filled-in image for each of the filled-in images to obtain a group of detection results for the filled-in images; a first removal unit that performs a first removal process on a detection result set consisting of the detection result for the target image and the detection result group for the filled-in images, and obtains the detection result set after unnecessary bounding boxes are removed as a first result set; a second removal unit that performs a second removal process on the first result set to obtain a second result set, which is the first result set after extra bounding boxes that were not removed in the first removal process have been removed, as a final result of the object detection for the target image; Equipped with. Effect of the Invention

[0007] According to the present disclosure, the effect of adversarial sample attacks can be reduced. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a configuration diagram of an object detection device 100 according to a first embodiment. [Diagram 2] FIG. 2 is a functional configuration diagram of the object detection device 100 according to the first embodiment. [Diagram 3] 3 is a flowchart of an object detection method according to the first embodiment. [Figure 4] 13 is a flowchart of step S150 in the first embodiment. [Diagram 5]13 is a flowchart of step S160 in the first embodiment. [Figure 6] FIG. 2 is a hardware configuration diagram of the object detection device 100 according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In the embodiments and drawings, the same or corresponding elements are denoted by the same reference numerals. Descriptions of elements denoted by the same reference numerals as those of elements described above will be omitted or simplified as appropriate. Arrows in the drawings primarily indicate data flow or processing flow.

[0010] Embodiment 1 The object detection device 100 will be described with reference to FIGS.

[0011] ***Configuration Description*** The configuration of an object detection device 100 will be described with reference to FIG. The object detection device 100 is a computer including hardware such as a processor 101, a memory 102, an auxiliary storage device 103, a communication device 104, and an input / output interface 105. These pieces of hardware are connected to each other via signal lines.

[0012] The processor 101 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 101 is a CPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit.

[0013] The memory 102 is a volatile or non-volatile storage device. The memory 102 is also called a primary storage device or a main memory. For example, the memory 102 is a RAM. Data stored in the memory 102 is saved in the secondary storage device 103 as necessary. RAM is an abbreviation for Random Access Memory.

[0014] The auxiliary storage device 103 is a non-volatile storage device. For example, the auxiliary storage device 103 is a ROM, a HDD, a flash memory, or a combination of these. Data stored in the auxiliary storage device 103 is loaded into the memory 102 as needed. ROM is an abbreviation for Read Only Memory. HDD is an abbreviation for Hard Disk Drive.

[0015] The communication device 104 is a receiver and a transmitter. For example, the communication device 104 is a communication chip or a NIC. The communication of the object detection device 100 is performed using the communication device 104. NIC is an abbreviation for Network Interface Card.

[0016] The input / output interface 105 is a port to which an input device and an output device are connected. For example, the input / output interface 105 is a USB terminal, the input device is a keyboard and a mouse, and the output device is a display. Input and output of the object detection device 100 is performed using the input / output interface 105. USB is an abbreviation for Universal Serial Bus.

[0017] The object detection device 100 includes components such as a reception unit 110, a detection unit 120, a processing unit 130, an integration unit 140, and an output unit 150. These components are realized by software. The detection unit 120 includes a first detection unit 121 and a second detection unit 122 . The integration unit 140 includes a first removal unit 141 and a second removal unit 142 .

[0018] The auxiliary storage device 103 stores an object detection program for causing a computer to function as the reception unit 110, the detection unit 120, the processing unit 130, the integration unit 140, and the output unit 150. The object detection program is loaded into the memory 102 and executed by the processor 101. The auxiliary storage device 103 further stores an OS. At least a part of the OS is loaded into the memory 102 and executed by the processor 101. The processor 101 executes an object detection program while executing the OS. OS is an abbreviation for Operating System.

[0019] Input and output data of the object detection program are stored in the storage unit 190. The memory 102 functions as the storage unit 190. However, a storage device such as the auxiliary storage device 103, a register in the processor 101, or a cache memory in the processor 101 may function as the storage unit 190 instead of the memory 102 or together with the memory 102.

[0020] The object detection program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or a flash memory.

[0021] FIG. 2 shows the functional configuration of the object detection device 100. The processing and input / output data of each element of the object detection device 100 will be described later.

[0022] ***Explanation of Operation*** The operation procedure of the object detection device 100 corresponds to an object detection method, and also corresponds to a processing procedure according to an object detection program.

[0023] The object detection method will be described with reference to FIG. In step S110, the receiving unit 110 receives the target image 191. For example, a user inputs a target image 191 into the object detection device 100 , and the reception unit 110 receives the input target image 191 . The target image 191 is data of an image on which object detection is performed.

[0024] In step S120, the first detection unit 121 performs object detection on the target image 191. As a result, a detection result 192 is calculated.

[0025] The object detection is performed using an object detector. For example, the object detector corresponds to a trained model, is realized by software, and is stored in the storage unit 190 in advance. An object detector is constructed using, for example, a neural network. Methods such as YOLO, SSD, or Faster R-CNN are used for object detectors. YOLO stands for You Only Look Once. SSD stands for Single Shot MultiBox Detector. CNN stands for Convolutional Neural Network. The first detection unit 121 performs object detection on the target image 191 by inputting the target image 191 and operating an object detector.

[0026] A detection result 192 is a result of object detection on the target image 191 (object detection result). The object detection result indicates one or more pairs of a bounding box, a score value, and a label. A bounding box is a rectangular area that encloses an object detected in an image and is represented using coordinate values ​​in the image. The score value is a probability that represents the confidence in the bounding box. The label represents the type of object within the bounding box. One or more bounding boxes shown in the object detection result are called a bounding box group. One or more score values ​​indicated in the object detection result are referred to as a score value group. One or more labels indicated in an object detection result are called a label set.

[0027] The object detection result indicates a set of bounding boxes, a set of scores for the set of bounding boxes, and a set of labels for the set of bounding boxes.

[0028] In step S130, the processing unit 130 fills in the target bounding box of the target image 191 for each target bounding box selected from the group of bounding boxes of the target image 191 to generate a filled-in image. As a result, a group of filled-in images 193 is obtained. The bounding boxes of the target image 191 are the bounding boxes shown in the detection result 192. The filled image group 193 is one or more filled images. A filled image is an object image 191 with an object bounding box filled in. The object bounding box is filled with a solid color. The target bounding box is a bounding box that corresponds to each score value included in a predetermined range, which is a predetermined range of score values.

[0029] The target bounding box is selected as follows: First, the processing unit 130 selects a score value included in a predetermined range from the score value group of the target image 191. The score value group of the target image 191 is a score value group indicated in the detection result 192. Then, for each selected score value, the processing unit 130 selects a bounding box corresponding to the selected score value from the group of bounding boxes of the target image 191. The selected bounding box is the target bounding box.

[0030] In step S140, the second detection unit 122 performs object detection on each filled-in image in the filled-in image group 193. As a result, a detection result group 194 is obtained. The object detection performed in step S140 is the same as the object detection performed in step S120. The detection result group 194 is a detection result group for the filled-in images group 193, and includes object detection results for each filled-in image.

[0031] The collection of the detection result 192 and the detection result group 194 is referred to as a detection result set.

[0032] In step S150, the first removal unit 141 performs a first removal process on the detection result set. The first removal process is a process of removing redundant bounding boxes based on the area of ​​the intersection of the bounding boxes of each bounding box pair.

[0033] The first removal process corresponds to Non-Maximum Suppression (NMS), which is also performed in normal object detection.

[0034] Specifically, the first removal process is as follows. A first reference bounding box is selected from the detection result set in descending order of score value. Each time a first reference bounding box is selected, each of the bounding boxes in the detection result set other than the first reference bounding box is set as a first comparison bounding box to calculate a first evaluation value, and if the first evaluation value satisfies a first removal condition, the first comparison bounding box is removed from the detection result set. The first evaluation value is a value obtained by dividing the area of ​​the intersection of the first reference bounding box and the first comparison bounding box by the area of ​​the union of the first reference bounding box and the first comparison bounding box.

[0035] The procedure of step S150 will be described with reference to FIG. In step S151, the first removal unit 141 selects, from the detection result set, the bounding box with the largest score value among the bounding boxes that have not been selected as the first reference bounding box.

[0036] The bounding box selected in step S151 is called the first reference bounding box.

[0037] In step S152, the first removal unit 141 selects one of the bounding boxes that has not been selected as a first comparison bounding box for the first reference bounding box from the detection result set. A bounding box other than the first reference bounding box is selected.

[0038] The bounding box selected in step S152 is referred to as a first comparison bounding box.

[0039] In step S153, the first removal unit 141 calculates a first evaluation value for the pair of the first reference bounding box and the first comparison bounding box. The first evaluation value becomes closer to 1 as the overlap between the first reference bounding box and the first comparison bounding box increases.

[0040] The first evaluation value is expressed as follows: IOU = (A∩B) / (A∪B) IOU is the first evaluation value. IOU is an abbreviation for Intersection Over Union. "A" means the first reference bounding box. "B" stands for the first comparison bounding box. (A∩B) is the area of ​​the intersection of the first reference bounding box and the first comparison bounding box. (A∪B) is the area of ​​the union of the first reference bounding box and the first comparison bounding box.

[0041] In step S154, the first removal unit 141 determines whether the first evaluation value satisfies the first removal condition. Specifically, the first removal unit 141 compares the first evaluation value with a first threshold and determines whether the first evaluation value is equal to or greater than the first threshold. If the first evaluation value is equal to or greater than the first threshold, the first evaluation value satisfies the first removal condition. The first threshold is a threshold for the first removal process and is determined in advance.

[0042] If the first evaluation value satisfies the first removal condition, the process proceeds to step S155. If the first evaluation value does not satisfy the first removal condition, the process proceeds to step S156.

[0043] In step S155, the first removal unit 141 removes the first comparison bounding box from the detection result set. This first comparison bounding box is an unnecessary bounding box.

[0044] In step S156, the first removal unit 141 determines whether or not there is an unselected first comparison bounding box in the detection result set. An unselected first comparison bounding box is a bounding box that has not been selected as a first comparison bounding box for the first reference bounding box.

[0045] If an unselected first comparison bounding box is in the detection result set, the process proceeds to step S152. If there is no unselected first comparison bounding box in the detection result set, the process proceeds to step S157.

[0046] In step S157, the first removal unit 141 determines whether or not an unselected first reference bounding box is present in the detection result set. An unselected first reference bounding box is a bounding box that has not been selected as a first reference bounding box.

[0047] If an unselected first reference bounding box is present in the detection result set, the process proceeds to step S151. If there is no unselected first reference bounding box in the detection result set, step S150 ends.

[0048] Returning to FIG. 3, the explanation will be continued. Step S150 results in a first result set 195. A first result set 195 is the detection result set after the redundant bounding boxes have been removed by a first removal process.

[0049] In step S160, the second removal unit 142 performs a second removal process on the first result set 195. The second removal process is a process of removing redundant bounding boxes based on the area of ​​the intersection of the bounding boxes of each bounding box pair.

[0050] Specifically, the second removal process is as follows. From the first result set 195, the second reference bounding boxes are selected in descending order of area. Each time a second reference bounding box is selected, a second evaluation value is calculated by treating each of the bounding boxes in the first result set 195 other than the second reference bounding box as a second comparison bounding box, and if the second evaluation value satisfies a second removal condition, the second comparison bounding box is removed from the first result set 195. The second evaluation value is a value obtained by dividing the area of ​​the intersection of the second reference bounding and the second comparison bounding box by the area of ​​the second comparison bounding box.

[0051] The procedure of step S160 will be described with reference to FIG. In step S161, the second removal unit 142 selects, from the first result set 195, the bounding box with the largest area among the bounding boxes that have not been selected as the second reference bounding box.

[0052] The bounding box selected in step S161 is called the second reference bounding box.

[0053] In step S162, the second removal unit 142 selects one of the bounding boxes that has not been selected as the second comparison bounding box for the second reference bounding box from the first result set 195. A bounding box other than the second reference bounding box is selected.

[0054] The bounding box selected in step S162 is referred to as a second comparison bounding box.

[0055] In step S163, the second removal unit 142 calculates a second evaluation value for the pair of the second reference bounding box and the second comparison bounding box. The second evaluation value approaches 1 as the proportion of the entire second comparison bounding box that is included in the second reference bounding box increases.

[0056] The second evaluation value is expressed as follows. IOA = (C∩D) / (D) IOA is the second evaluation value. "C" means the second reference bounding box. "D" stands for the second comparison bounding box. (C∩D) is the area of ​​the intersection of the second reference bounding box and the second comparison bounding box. (D) is the area of ​​the second comparison bounding box.

[0057] In step S164, the second removal unit 142 determines whether the second evaluation value satisfies a second removal condition. Specifically, the second removal unit 142 compares the second evaluation value with a second threshold and determines whether the second evaluation value is equal to or greater than the second threshold. If the second evaluation value is equal to or greater than the second threshold, the second evaluation value satisfies the second removal condition. The second threshold is a threshold for the second removal process and is determined in advance.

[0058] If the second evaluation value satisfies the second removal condition, the process proceeds to step S165. If the second evaluation value does not satisfy the second removal condition, the process proceeds to step S166.

[0059] In step S165, the second removal unit 142 removes the second comparison bounding box from the first result set 195. This second comparison bounding box is an unnecessary bounding box.

[0060] In step S166, the second removal unit 142 determines whether there is an unselected second comparison bounding box in the first result set 195. An unselected second comparison bounding box is a bounding box that has not been selected as a second comparison bounding box for the second reference bounding box.

[0061] If the unselected second comparison bounding box is in first result set 195, processing proceeds to step S162. If there is no unselected second comparison bounding box in first result set 195, processing proceeds to step S167.

[0062] In step S167, the second removal unit 142 determines whether or not there is an unselected second reference bounding box in the first result set 195. An unselected second reference bounding box is a bounding box that has not been selected as a second reference bounding box.

[0063] If an unselected second reference bounding box is in first result set 195, processing proceeds to step S161. If there are no unselected second reference bounding boxes in the first result set 195, step S160 ends.

[0064] Returning to FIG. 3, the explanation will be continued. Step S160 results in a second result set 196. A second result set 196 is the first result set 195 after the extraneous bounding boxes that were not removed in the first removal process have been removed by the second removal process.

[0065] In step S170, the output unit 150 outputs the second result set 196 as the object detection result 197. The object detection result 197 is the final result of object detection on the target image 191 .

[0066] For example, the output unit 150 displays the object detection result 197 on a display.

[0067] ***Features of the First Embodiment*** In the first embodiment, in order to deal with the adversarial sample patch attack against object detection, the position of the adversarial sample patch (target bounding box) is estimated based on the score value of the bounding box output from the object detector for the input image, and the estimated position is filled in. This reduces the effect of the attack. In the first embodiment, a first removal process and a second removal process are performed on the detection results of the images before and after filling to remove unnecessary bounding boxes, and the remaining bounding boxes are output as correct bounding boxes in which attacks have been neutralized.

[0068] ***Advantages of the First Embodiment*** According to the first embodiment, even if an attack is made to evade object detection by using adversarial sample patches, it is possible to nullify the effect of the attack and output a bounding box that should be output in object detection.

[0069] ***Addition to the first embodiment*** The hardware configuration of the object detection device 100 will be described with reference to FIG. The object detection device 100 includes a processing circuit 109 . The processing circuit 109 is hardware that realizes the reception unit 110, the detection unit 120, the processing unit 130, the integration unit 140, and the output unit 150. The processing circuitry 109 may be dedicated hardware, or may be a processor 101 that executes a program stored in the memory 102 .

[0070] When the processing circuitry 109 is dedicated hardware, the processing circuitry 109 may be, for example, a single circuit, a multiple circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field Programmable Gate Array.

[0071] The object detection device 100 may include a plurality of processing circuits instead of the processing circuit 109 .

[0072] In the processing circuit 109, some functions may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware.

[0073] Thus, the functions of the object detection device 100 can be realized by hardware, software, firmware, or a combination of these.

[0074] The first embodiment is an example of a preferred embodiment, and is not intended to limit the technical scope of the present disclosure. The first embodiment may be implemented partially or in combination with other embodiments. The procedure described using the flow chart, etc. may be modified as appropriate.

[0075] The "part" of each element of the object detection device 100 may be interpreted as a "process," a "step," a "circuit," or a "circuitry." [Explanation of symbols]

[0076] 100 object detection device, 101 processor, 102 memory, 103 auxiliary storage device, 104 communication device, 105 input / output interface, 109 processing circuit, 110 reception unit, 120 detection unit, 121 first detection unit, 122 second detection unit, 130 processing unit, 140 integration unit, 141 first removal unit, 142 second removal unit, 150 output unit, 190 memory unit, 191 target image, 192 detection result, 193 filled-in image group, 194 detection result group, 195 first result set, 196 second result set, 197 object detection result.

Claims

1. a first detection unit that performs object detection on a target image and calculates a detection result indicating a bounding box group; a processing unit that obtains a filled-in image group by filling in the target bounding box of the target image for each target bounding box selected from the bounding box group of the target image to generate a filled-in image; a second detection unit that performs the object detection on the filled-in image for each of the filled-in images to obtain a group of detection results for the filled-in images; a first removal unit that performs a first removal process on a detection result set consisting of the detection result for the target image and the detection result group for the filled-in images, and obtains, as a first result set, the detection result set from which unnecessary bounding boxes have been removed; a second removal unit that performs a second removal process on the first result set to obtain a second result set, which is the first result set after extra bounding boxes that were not removed in the first removal process have been removed, as a final result of the object detection for the target image; An object detection device comprising:

2. Each of the first removal process and the second removal process is a process for removing the redundant bounding boxes based on the area of ​​the intersection of the bounding boxes of each bounding box pair. The object detection device according to claim 1 .

3. Each detection result further indicates a set of score values ​​for the set of bounding boxes. The object detection device according to claim 1 or 2.

4. The first removal process includes: Select a first reference bounding box from the detection result set in descending order of score value; each time the first reference bounding box is selected, each of the bounding boxes in the detection result set other than the first reference bounding box is set as a first comparison bounding box, an area of ​​a product of the first reference bounding box and the first comparison bounding box is divided by an area of ​​a union of the first reference bounding box and the first comparison bounding box to calculate a first evaluation value, and if the first evaluation value satisfies a first removal condition, the first comparison bounding box is removed from the detection result set; The second removal process includes: Selecting second reference bounding boxes from the first result set in descending order of area; Each time the second reference bounding box is selected, each of the bounding boxes in the first result set other than the second reference bounding box is set as a second comparison bounding box, an area of ​​the intersection of the second reference bounding box and the second comparison bounding box is divided by an area of ​​the second comparison bounding box to calculate a second evaluation value, and if the second evaluation value satisfies a second removal condition, the second comparison bounding box is removed from the first result set. The object detection device according to claim 3 .

5. The processing unit selects, as the target bounding box, a bounding box corresponding to each of the score values ​​included in a predetermined range. The object detection device according to claim 3 .

6. Performing object detection on the target image and calculating a detection result indicating a group of bounding boxes; For each target bounding box selected from the group of bounding boxes of the target image, fill the target bounding box of the target image to generate a filled-in image, thereby obtaining a group of filled-in images; performing the object detection on the filled-in image for each of the filled-in images to obtain a group of detection results for the group of filled-in images; performing a first removal process on a detection result set consisting of the detection result for the target image and the detection result set for the filled-in images to obtain the detection result set after unnecessary bounding boxes have been removed as a first result set; A second removal process is performed on the first result set, and a second result set, which is the first result set after extra bounding boxes that were not removed in the first removal process have been removed, is obtained as a final result of the object detection for the target image. Object detection methods.

7. a first detection process for performing object detection on a target image and calculating a detection result indicating a bounding box group; a processing process for obtaining a filled-in image group by filling in the target bounding box of the target image for each target bounding box selected from the bounding box group of the target image to generate a filled-in image; a second detection process for performing the object detection on the filled-in image for each of the filled-in images to obtain a group of detection results for the filled-in images; a first removal process for performing a first removal process on a detection result set including the detection result for the target image and the detection result group for the filled-in images to obtain the detection result set after unnecessary bounding boxes are removed as a first result set; a second removal process for performing a second removal process on the first result set to obtain a second result set, which is the first result set after removing extra bounding boxes that were not removed in the first removal process, as a final result of the object detection for the target image; An object detection program for causing a computer to execute the above.

Images