Object detection device, object detection method, and object detecting program

The object detection device addresses the vulnerability of deep learning systems to adversarial attacks by filling in bounding boxes and applying removal processes, resulting in robust and accurate object detection outcomes.

WO2024231970A1PCT designated stage expired Publication Date: 2024-11-14MITSUBISHI ELECTRIC CORP

Patent Information

Application Number
PCT/JP2023/017220
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-08
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Deep learning-based object detection systems are vulnerable to adversarial sample attacks, which falsify classification results by adding perturbations to input data, leading to inaccurate object detection.

Method used

An object detection device and method that performs object detection on an image, fills in selected bounding boxes, and applies two removal processes to eliminate redundant bounding boxes, thereby reducing the effectiveness of adversarial sample attacks.

Benefits of technology

The proposed solution effectively nullifies the impact of adversarial sample attacks, ensuring accurate object detection by removing extra bounding boxes and enhancing the robustness of the system against hostile samples.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023017220_14112024_PF_FP_ABST
    Figure JP2023017220_14112024_PF_FP_ABST
Patent Text Reader

Abstract

A first detection unit (121) performs object detection on a target image (191) and calculates a detection result (192). A processing unit (130) fills each of target bounding boxes of the target image to obtain a filled image group (193). A second detection unit (122) performs the object detection on each filled image to obtain a detection result group (194). A first removal unit (141) performs first removal processing on a detection result set to obtain a first result set (195). A second removal unit (142) performs second removal processing on the first result set to obtain a second result set (196) as an object detection result (197).
Need to check novelty before this filing date? Find Prior Art

Description

OBJECT DETECTION DEVICE, OBJECT DETECTION METHOD, AND OBJECT DETECTION PROGRAM

[0001] The present disclosure relates to a technology for detecting an object captured in an image.

[0002] In the object detection task, the position of each object in an input image is indicated by a bounding box, and the type of each object is indicated by a label. In recent years, deep learning methods using neural networks have achieved very high accuracy in object detection tasks.

[0003] An image classifier (e.g., a multi-class classifier) ​​is constructed by deep learning. Non-Patent Document 1 describes an adversarial example attack that adds perturbations to input data to cause erroneous classification results by the multi-class classifier.

[0004] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow and Rob Fergus: Intriguing properties of neural networks, in International Conference on Learning Representations (ICLR) (2014)

[0005] The present disclosure aims to reduce the effects of adversarial sample attacks.

[0006] The object detection device of the present disclosure includes: a first detection unit that performs object detection on a target image and calculates a detection result indicating a group of bounding boxes; a processing unit that obtains a group of filled-in images by filling in each target bounding box selected from the group of bounding boxes of the target image to generate a filled-in image; a second detection unit that performs object detection on each filled-in image to obtain a group of detection results for the group of filled-in images; a first removal unit that performs a first removal process on a detection result set consisting of the detection results for the target image and the detection result set for the group of filled-in images, to obtain the detection result set after extraneous bounding boxes have been removed, as a first result set; and a second removal unit that performs a second removal process on the first result set, to obtain a second result set that is the first result set after extraneous bounding boxes that were not removed in the first removal process have been removed, as a final result of the object detection for the target image.

[0007] According to the present disclosure, the effect of adversarial sample attacks can be reduced.

[0008] FIG. 1 is a configuration diagram of an object detection device 100 according to embodiment 1. FIG. 2 is a functional configuration diagram of the object detection device 100 according to embodiment 1. FIG. 3 is a flowchart of an object detection method according to embodiment 1. FIG. 4 is a flowchart of step S150 according to embodiment 1. FIG. 5 is a flowchart of step S160 according to embodiment 1. FIG. 6 is a hardware configuration diagram of the object detection device 100 according to embodiment 1.

[0009] In the embodiments and drawings, the same or corresponding elements are denoted by the same reference numerals. The description of elements denoted by the same reference numerals as those already described will be omitted or simplified as appropriate. Arrows in the drawings primarily indicate the flow of data or the flow of processing.

[0010] First Embodiment An object detection device 100 will be described with reference to FIGS.

[0011] ***Description of Configuration*** The configuration of the object detection device 100 will be described with reference to Fig. 1. The object detection device 100 is a computer equipped with hardware such as a processor 101, a memory 102, an auxiliary storage device 103, a communication device 104, and an input / output interface 105. These pieces of hardware are connected to one another via signal lines.

[0012] The processor 101 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 101 is a CPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit.

[0013] The memory 102 is a volatile or non-volatile storage device. The memory 102 is also called a primary storage device or a main memory. For example, the memory 102 is a RAM. Data stored in the memory 102 is saved in the secondary storage device 103 as needed. RAM is an abbreviation for Random Access Memory.

[0014] The auxiliary storage device 103 is a non-volatile storage device. For example, the auxiliary storage device 103 is a ROM, a HDD, a flash memory, or a combination of these. Data stored in the auxiliary storage device 103 is loaded into the memory 102 as needed. ROM is an abbreviation for Read Only Memory. HDD is an abbreviation for Hard Disk Drive.

[0015] The communication device 104 is a receiver and a transmitter. For example, the communication device 104 is a communication chip or a NIC. Communication of the object detection device 100 is performed using the communication device 104. NIC is an abbreviation for Network Interface Card.

[0016] The input / output interface 105 is a port to which an input device and an output device are connected. For example, the input / output interface 105 is a USB terminal, the input devices are a keyboard and a mouse, and the output device is a display. Input and output of the object detection device 100 is performed using the input / output interface 105. USB is an abbreviation for Universal Serial Bus.

[0017] The object detection device 100 includes elements such as a reception unit 110, a detection unit 120, a processing unit 130, an integration unit 140, and an output unit 150. These elements are implemented by software. The detection unit 120 includes a first detection unit 121 and a second detection unit 122. The integration unit 140 includes a first removal unit 141 and a second removal unit 142.

[0018] The auxiliary storage device 103 stores an object detection program for causing the computer to function as the reception unit 110, the detection unit 120, the processing unit 130, the integration unit 140, and the output unit 150. The object detection program is loaded into the memory 102 and executed by the processor 101. The auxiliary storage device 103 also stores an OS. At least a portion of the OS is loaded into the memory 102 and executed by the processor 101. The processor 101 executes the object detection program while running the OS. OS is an abbreviation for Operating System.

[0019] Input and output data of the object detection program is stored in the storage unit 190. The memory 102 functions as the storage unit 190. However, a storage device such as the auxiliary storage device 103, a register in the processor 101, or a cache memory in the processor 101 may function as the storage unit 190 instead of or together with the memory 102.

[0020] The object detection program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or a flash memory.

[0021] 2 shows the functional configuration of the object detection device 100. The processing of each element of the object detection device 100 and the input / output data will be described later.

[0022] ***Description of Operation*** The operation procedure of the object detection device 100 corresponds to an object detection method. Also, the operation procedure of the object detection device 100 corresponds to a processing procedure according to an object detection program.

[0023] The object detection method will be described with reference to Fig. 3. In step S110, the reception unit 110 receives a target image 191. For example, a user inputs the target image 191 to the object detection device 100, and the reception unit 110 receives the input target image 191. The target image 191 is image data for which object detection is performed.

[0024] In step S120, the first detection unit 121 performs object detection on the target image 191. As a result, a detection result 192 is calculated.

[0025] Object detection is performed using an object detector. For example, the object detector corresponds to a trained model, is realized by software, and is stored in advance in the storage unit 190. The object detector is constructed using, for example, a neural network. Methods such as YOLO, SSD, or Faster R-CNN are used for the object detector. YOLO is an abbreviation for You Only Look Once. SSD is an abbreviation for Single Shot MultiBox Detector. CNN is an abbreviation for Convolutional Neural Network. The first detection unit 121 performs object detection on the target image 191 by inputting the target image 191 and operating the object detector.

[0026] The detection result 192 is the result of object detection for the target image 191 (object detection result). The object detection result indicates one or more pairs of a bounding box, a score value, and a label. The bounding box is a rectangular area that surrounds an object detected from an image, and is indicated using coordinate values ​​in the image. The score value is a probability that indicates the confidence in the bounding box. The label indicates the type of object within the bounding box. One or more bounding boxes indicated in the object detection result are referred to as a bounding box group. One or more score values ​​indicated in the object detection result are referred to as a score value group. One or more labels indicated in the object detection result are referred to as a label group.

[0027] The object detection result indicates a group of bounding boxes, a group of scores for the group of bounding boxes, and a group of labels for the group of bounding boxes.

[0028] In step S130, the processing unit 130 fills in the target bounding box of the target image 191 for each target bounding box selected from the group of bounding boxes of the target image 191 to generate a filled-in image. This results in a group of filled-in images 193. The group of bounding boxes of the target image 191 is the group of bounding boxes shown in the detection result 192. The group of filled-in images 193 is one or more filled-in images. A filled-in image is the target image 191 in which the target bounding box is filled in. The target bounding box is filled in with a single color. The target bounding box is a bounding box corresponding to each score value included in a predetermined range. The predetermined range is a fixed range determined in advance for the score value.

[0029] The target bounding box is selected as follows: First, the processing unit 130 selects a score value included in a predetermined range from the group of score values ​​of the target image 191. The group of score values ​​of the target image 191 is the group of score values ​​indicated in the detection result 192. Then, for each selected score value, the processing unit 130 selects a bounding box corresponding to the selected score value from the group of bounding boxes of the target image 191. The selected bounding box is the target bounding box.

[0030] In step S140, the second detection unit 122 performs object detection on each filled-in image in the filled-in image group 193. This results in a detection result group 194. The object detection performed in step S140 is the same as the object detection performed in step S120. The detection result group 194 is a detection result group for the filled-in image group 193 and includes object detection results for each filled-in image.

[0031] The set of the detection result 192 and the detection result group 194 is called a detection result set.

[0032] In step S150, the first removal unit 141 performs a first removal process on the detection result set. The first removal process is a process of removing redundant bounding boxes based on the area of ​​the intersection of the bounding boxes in each bounding box pair.

[0033] The first removal process corresponds to Non-Maximum Suppression (NMS), which is also performed in normal object detection.

[0034] Specifically, the first removal process is as follows: A first reference bounding box is selected from the detection result set in descending order of score value. Each time a first reference bounding box is selected, a first evaluation value is calculated using each of the bounding boxes in the detection result set other than the first reference bounding box as a first comparison bounding box, and if the first evaluation value satisfies a first removal condition, the first comparison bounding box is removed from the detection result set. The first evaluation value is a value obtained by dividing the area of ​​the intersection of the first reference bounding box and the first comparison bounding box by the area of ​​the union of the first reference bounding box and the first comparison bounding box.

[0035] The procedure of step S150 will be described with reference to Fig. 4. In step S151, the first removal unit 141 selects, from the detection result set, the bounding box with the largest score value among the bounding boxes that have not been selected as the first reference bounding box.

[0036] The bounding box selected in step S151 is referred to as a first reference bounding box.

[0037] In step S152, the first removal unit 141 selects one of the bounding boxes that has not been selected as a first comparison bounding box for the first reference bounding box from the detection result set. The selected bounding box is different from the first reference bounding box.

[0038] The bounding box selected in step S152 is referred to as a first comparison bounding box.

[0039] In step S153, the first removal unit 141 calculates a first evaluation value for the pair of the first reference bounding box and the first comparison bounding box. The first evaluation value becomes closer to 1 as the overlap between the first reference bounding box and the first comparison bounding box increases.

[0040] The first evaluation value is expressed as follows: IOU = (A∩B) / (A∪B) IOU is the first evaluation value. IOU is an abbreviation for Intersection Over Union. "A" refers to the first reference bounding box. "B" refers to the first comparison bounding box. (A∩B) is the area of ​​the intersection of the first reference bounding box and the first comparison bounding box. (A∪B) is the area of ​​the union of the first reference bounding box and the first comparison bounding box.

[0041] In step S154, the first removal unit 141 determines whether the first evaluation value satisfies the first removal condition. Specifically, the first removal unit 141 compares the first evaluation value with a first threshold and determines whether the first evaluation value is equal to or greater than the first threshold. If the first evaluation value is equal to or greater than the first threshold, the first evaluation value satisfies the first removal condition. The first threshold is a threshold for the first removal process and is determined in advance.

[0042] If the first evaluation value satisfies the first removal condition, the process proceeds to step S155. If the first evaluation value does not satisfy the first removal condition, the process proceeds to step S156.

[0043] In step S155, the first removal unit 141 removes the first comparison bounding box from the detection result set. This first comparison bounding box is an unnecessary bounding box.

[0044] In step S156, the first removal unit 141 determines whether there is an unselected first comparison bounding box in the detection result set. An unselected first comparison bounding box is a bounding box that has not been selected as a first comparison bounding box for the first reference bounding box.

[0045] If the unselected first comparison bounding box is in the detection result set, the process proceeds to step S152. If the unselected first comparison bounding box is not in the detection result set, the process proceeds to step S157.

[0046] In step S157, the first removal unit 141 determines whether there is an unselected first reference bounding box in the detection result set. An unselected first reference bounding box is a bounding box that has not been selected as a first reference bounding box.

[0047] If an unselected first reference bounding box is present in the detection result set, the process proceeds to step S151. If an unselected first reference bounding box is not present in the detection result set, step S150 ends.

[0048] 3, the description continues. In step S150, a first result set 195 is obtained. The first result set 195 is a detection result set after unnecessary bounding boxes have been removed by the first removal process.

[0049] In step S160, the second removal unit 142 performs a second removal process on the first result set 195. The second removal process is a process of removing redundant bounding boxes based on the area of ​​the intersection of the bounding boxes in each bounding box pair.

[0050] Specifically, the second removal process is as follows: Second reference bounding boxes are selected from the first result set 195 in descending order of area. Each time a second reference bounding box is selected, a second evaluation value is calculated using each of the bounding boxes in the first result set 195 other than the second reference bounding box as a second comparison bounding box, and if the second evaluation value satisfies the second removal condition, the second comparison bounding box is removed from the first result set 195. The second evaluation value is a value obtained by dividing the area of ​​the intersection of the second reference bounding box and the second comparison bounding box by the area of ​​the second comparison bounding box.

[0051] The procedure of step S160 will be described with reference to Fig. 5. In step S161, the second removal unit 142 selects, from the first result set 195, the bounding box with the largest area among the bounding boxes that have not been selected as the second reference bounding box.

[0052] The bounding box selected in step S161 is referred to as a second reference bounding box.

[0053] In step S162, the second removal unit 142 selects one of the bounding boxes that has not been selected as a second comparison bounding box for the second reference bounding box from the first result set 195. A bounding box other than the second reference bounding box is selected.

[0054] The bounding box selected in step S162 is referred to as a second comparison bounding box.

[0055] In step S163, the second removal unit 142 calculates a second evaluation value for the pair of the second reference bounding box and the second comparison bounding box. The second evaluation value approaches 1 as the proportion of the entire second comparison bounding box that is included in the second reference bounding box increases.

[0056] The second evaluation value is expressed as follows: IOA = (C∩D) / (D) IOA is the second evaluation value. "C" means the second reference bounding box. "D" means the second comparison bounding box. (C∩D) is the area of ​​the intersection of the second reference bounding box and the second comparison bounding box. (D) is the area of ​​the second comparison bounding box.

[0057] In step S164, the second removal unit 142 determines whether the second evaluation value satisfies the second removal condition. Specifically, the second removal unit 142 compares the second evaluation value with a second threshold and determines whether the second evaluation value is equal to or greater than the second threshold. If the second evaluation value is equal to or greater than the second threshold, the second evaluation value satisfies the second removal condition. The second threshold is a threshold for the second removal process and is determined in advance.

[0058] If the second evaluation value satisfies the second removal condition, the process proceeds to step S165. If the second evaluation value does not satisfy the second removal condition, the process proceeds to step S166.

[0059] In step S165, the second removal unit 142 removes the second comparison bounding box from the first result set 195. This second comparison bounding box is an unnecessary bounding box.

[0060] In step S166, the second removal unit 142 determines whether there are any unselected second comparison bounding boxes in the first result set 195. An unselected second comparison bounding box is a bounding box that has not been selected as a second comparison bounding box for the second reference bounding box.

[0061] If the unselected second comparison bounding box is in first result set 195, processing proceeds to step S162. If the unselected second comparison bounding box is not in first result set 195, processing proceeds to step S167.

[0062] In step S167, the second removal unit 142 determines whether there is an unselected second reference bounding box in the first result set 195. An unselected second reference bounding box is a bounding box that has not been selected as a second reference bounding box.

[0063] If an unselected second reference bounding box is in the first result set 195, the process proceeds to step S161. If an unselected second reference bounding box is not in the first result set 195, step S160 ends.

[0064] 3, the description continues. Step S160 obtains a second result set 196. The second result set 196 is the first result set 195 after the extra bounding boxes that were not removed in the first removal process have been removed in the second removal process.

[0065] In step S170, the output unit 150 outputs the second result set 196 as an object detection result 197. The object detection result 197 is the final result of object detection for the target image 191.

[0066] For example, the output unit 150 displays the object detection result 197 on a display.

[0067] ***Features of First Embodiment*** In the first embodiment, to counter adversarial sample patch attacks on object detection, the position of the adversarial sample patch (target bounding box) is estimated based on the score value of the bounding box output from an object detector for an input image, and the estimated position is filled in. This reduces the effectiveness of the attack. In the first embodiment, a first removal process and a second removal process are performed on the detection results of the image before filling and the image group after filling to remove redundant bounding boxes, and the remaining bounding boxes are output as correct bounding boxes that have been neutralized by the attack.

[0068] ***Effects of Embodiment 1*** According to embodiment 1, even if an attack is carried out using adversarial sample patches to evade object detection, the effect of the attack can be nullified and the bounding box that should be output by object detection can be output.

[0069] *** Supplementary Note to First Embodiment *** The hardware configuration of object detection device 100 will be described with reference to Fig. 6. Object detection device 100 includes processing circuitry 109. Processing circuitry 109 is hardware that implements reception unit 110, detection unit 120, processing unit 130, integrating unit 140, and output unit 150. Processing circuitry 109 may be dedicated hardware, or may be processor 101 that executes a program stored in memory 102.

[0070] When the processing circuit 109 is dedicated hardware, the processing circuit 109 may be, for example, a single circuit, a multiple circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field Programmable Gate Array.

[0071] The object detection device 100 may include a plurality of processing circuits that replace the processing circuit 109 .

[0072] In the processing circuit 109, some functions may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware.

[0073] In this way, the functions of the object detection device 100 can be realized by hardware, software, firmware, or a combination of these.

[0074] The first embodiment is an example of a preferred embodiment and is not intended to limit the technical scope of the present disclosure. The first embodiment may be implemented in part or in combination with other embodiments. The procedures described using flowcharts and the like may be modified as appropriate.

[0075] The "part" of each element of the object detection device 100 may be read as a "process," a "step," a "circuit," or a "circuitry."

[0076] 100 Object detection device, 101 Processor, 102 Memory, 103 Auxiliary storage device, 104 Communication device, 105 Input / output interface, 109 Processing circuit, 110 Reception unit, 120 Detection unit, 121 First detection unit, 122 Second detection unit, 130 Processing unit, 140 Integration unit, 141 First removal unit, 142 Second removal unit, 150 Output unit, 190 Storage unit, 191 Target image, 192 Detection result, 193 Filled image group, 194 Detection result group, 195 First result set, 196 Second result set, 197 Object detection result.

Claims

1. An object detection device comprising: a first detection unit that performs object detection on a target image and calculates a detection result indicating a group of bounding boxes; a processing unit that obtains a group of filled-in images by filling in each target bounding box selected from the group of bounding boxes of the target image to generate a filled-in image; a second detection unit that performs the object detection on the filled-in image for each filled-in image to obtain a group of detection results for the filled-in image group; a first removal unit that performs a first removal process on a detection result set consisting of the detection results for the target image and the detection result set for the filled-in images group, to obtain the detection result set after unnecessary bounding boxes have been removed as a first result set; and a second removal unit that performs a second removal process on the first result set, to obtain a second result set which is the first result set after unnecessary bounding boxes not removed in the first removal process have been removed as a final result of the object detection for the target image.

2. The object detection device according to claim 1, wherein each of the first removal process and the second removal process is a process for removing the redundant bounding boxes based on the area of ​​the intersection of the bounding boxes of each bounding box pair.

3. The object detection device according to claim 1 or 2, wherein each detection result further indicates a set of score values ​​for the set of bounding boxes.

4. The first removal process selects a first reference bounding box from the detection result set in descending order of score value; each time a first reference bounding box is selected, each of the bounding boxes in the detection result set other than the first reference bounding box is set as a first comparison bounding box, and a first evaluation value is calculated by dividing the area of ​​the intersection of the first reference bounding box and the first comparison bounding box by the area of ​​the union of the first reference bounding box and the first comparison bounding box, and if the first evaluation value satisfies a first removal condition, the first comparison bounding box is removed from the detection result set; and the second removal process selects a second reference bounding box from the first result set in descending order of area; 4. The object detection device according to claim 3, wherein each time the second reference bounding box is selected, each of the bounding boxes in the first result set other than the second reference bounding box is set as a second comparison bounding box, an area of ​​a set intersection of the second reference bounding box and the second comparison bounding box is divided by an area of ​​the second comparison bounding box to calculate a second evaluation value, and if the second evaluation value satisfies a second removal condition, the second comparison bounding box is removed from the first result set.

5. The object detection device according to claim 3 or 4, wherein the processing unit selects, as the target bounding box, a bounding box corresponding to each score value included in a predetermined range.

6. An object detection method comprising: performing object detection on a target image to calculate a detection result indicating a group of bounding boxes; obtaining a group of filled images by filling in each target bounding box selected from the group of bounding boxes of the target image to generate a filled-in image; performing the object detection on each filled-in image to obtain a group of detection results for the group of filled-in images; performing a first removal process on a detection result set consisting of the detection results for the target image and the detection result set for the group of filled-in images to obtain a first result set, which is the detection result set after extra bounding boxes have been removed; performing a second removal process on the first result set to obtain a second result set, which is the first result set after extra bounding boxes not removed in the first removal process, as a final result of the object detection for the target image.

7. An object detection program for causing a computer to execute the following steps: a first detection process for performing object detection on a target image to calculate a detection result indicating a group of bounding boxes; a processing process for obtaining a group of filled-in images by filling in each target bounding box selected from the group of bounding boxes of the target image to generate a filled-in image; a second detection process for performing the object detection on each filled-in image to obtain a group of detection results for the group of filled-in images; a first removal process for performing a first removal process on a detection result set consisting of the detection results for the target image and the detection result set for the group of filled-in images to obtain a first result set, the detection result set after unnecessary bounding boxes have been removed; and a second removal process for performing a second removal process on the first result set to obtain a second result set, which is the first result set after unnecessary bounding boxes not removed in the first removal process, as a final result of the object detection for the target image.

Citation Information

Patent Citations

  • Image processor, image processing system, method for image processing, and program

    JP2018112991A

  • Target detection and target detection network training

    JP2021532435A

Cited By

  • Image processing device, image processing method, and image processing program

    WO2026018467A1