Object detection device and object detection method

Abstract

The object detection device (1) is configured to include a box generation unit (11) that detects an object in an input image and generates a bounding box surrounding the object, and a box selection unit (12) that, if multiple overlapping bounding boxes are generated by the box generation unit (11), selects one of the multiple bounding boxes based on the reliability of each of the bounding boxes. The object detection device (1) also includes an overlap calculation unit (13) that calculates the overlap between each of the bounding boxes generated by the box generation unit (11) and the bounding box selected by the box selection unit (12), and a contribution calculation unit (14) that calculates the contribution of pixels contributing to the detection of the object, among multiple pixels included in the input image, based on the overlap calculated by the overlap calculation unit (13).

Description

[Technical field]

[0001] The present disclosure relates to an object detection device and an object detection method. [Background technology]

[0002] There is an object detection device that detects an object by using an object detection model that detects an object in an input image. As such an object detection device, for example, Patent Document 1 discloses an object detection device including an adding unit and a deleting unit. The annotation unit detects an object in an input image and generates a bounding box surrounding the object. If the annotation unit generates multiple bounding boxes that overlap each other, the deletion unit performs a filtering process called NMS (Non-Maximum Suppression) to leave the most reliable bounding box among the multiple bounding boxes and delete the other bounding boxes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-118954 A Summary of the Invention [Problem to be solved by the invention]

[0004] For example, a deep learning-based image recognition model may be used as an object detection model. A deep learning-based image recognition model generally performs deep learning. For this reason, it may be difficult for a user to clearly understand the basis for detecting an object by an image recognition model. Therefore, if an object detection device can visualize pixels that contribute to the detection of an object among multiple pixels included in an input image, it is useful for a user to understand the basis for detecting an object. If it is possible to partially differentiate NMS, the gradient of the bounding box left by NMS for each of the overlapping bounding boxes can be calculated. Once the gradient of the left bounding box is calculated, it may be possible to calculate the contribution of each pixel in the input image that contributes to the detection of the object based on the gradient of the left bounding box. However, since the NMS is not expressed by a partial differential equation, it is difficult to partially differentiate the NMS. Therefore, the object detection device disclosed in Patent Document 1 has a problem that it cannot calculate the gradient of the bounding box left by the NMS for each of a plurality of bounding boxes that overlap each other.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an object detection device that can present the contribution of pixels that contribute to object detection among multiple pixels contained in an input image. [Means for solving the problem]

[0006] The object detection device according to the present disclosure includes a box generation unit that detects an object in an input image and generates one or more bounding boxes surrounding the object, and a box selection unit that, if multiple overlapping bounding boxes are generated by the box generation unit, selects one of the multiple bounding boxes based on the reliability of each of the bounding boxes.The object detection device also includes an overlap calculation unit that calculates an overlap between each of the bounding boxes generated by the box generation unit and the bounding box selected by the box selection unit, and a contribution calculation unit that calculates a contribution of pixels that contribute to the detection of the object among multiple pixels included in the input image based on the overlap calculated by the overlap calculation unit. Effect of the Invention

[0007] According to the present disclosure, it is possible to present the contribution of pixels that contribute to object detection among a plurality of pixels included in an input image. [Brief description of the drawings]

[0008] [Figure 1] 1 is a configuration diagram showing an object detection device 1 according to a first embodiment. [Diagram 2] 1 is a hardware configuration diagram showing hardware of an object detection device 1 according to a first embodiment. [Diagram 3] FIG. 1 is a hardware configuration diagram of a computer in the case where the object detection device 1 is realized by software, firmware, or the like. [Figure 4] 3 is a flowchart showing an object detection method which is a processing procedure of the object detection device 1. [Diagram 5] FIG. 2 is an explanatory diagram showing an example of an input image. [Figure 6] 1 is an explanatory diagram showing an example of bounding box generation by the box generation unit 11. FIG. [Figure 7] FIG. 11 is an explanatory diagram showing a calculation process of the overlap degree DSi,j between a bounding box Xi,j and a bounding box Xi′; [Figure 8] FIG. 2 is an explanatory diagram showing an example of an image of Grad-CAM. [Figure 9] FIG. 11 is a configuration diagram showing an object detection device 1 according to a second embodiment. [Figure 10] FIG. 11 is a hardware configuration diagram showing hardware of an object detection device 1 according to a second embodiment. [Figure 11] FIG. 11 is a configuration diagram showing an object detection device 1 according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In order to describe the present disclosure in more detail, embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0010] Embodiment 1 FIG. 1 is a configuration diagram showing an object detection device 1 according to the first embodiment. FIG. 2 is a hardware configuration diagram showing the hardware of the object detection device 1 according to the first embodiment. The object detection device 1 shown in FIG. 1 detects an object in an input image and generates one or more bounding boxes surrounding the object. Furthermore, object detection device 1 calculates the degree of contribution of pixels that contribute to the detection of an object, among a plurality of pixels included in the input image. The display device 2 displays an image indicating the contribution of each pixel calculated by the object detection device 1 on a display.

[0011] The object detection device 1 shown in FIG. 1 includes a box generation unit 11, a box selection unit 12, an overlapping degree calculation unit 13, a contribution degree calculation unit 14, and a display processing unit 15.

[0012] The box generating unit 11 is realized by, for example, a box generating circuit 21 shown in FIG. The box generation unit 11 uses, for example, a deep learning-based image recognition model as the object detection model. The object detection model is an object detection algorithm. In the object detection device 1 shown in Fig. 1, the box generation unit 11 uses a convolutional neural network (hereinafter referred to as "OD-CNN (Object Detection-Convolutional Neural Network)") as an image recognition model. The OD-CNN may be stored in an internal memory of the box generation unit 11, or may be stored in an external memory of the box generation unit 11. The box generating unit 11 acquires image data of an input image from the outside. The box generator 11 detects objects in an input image and generates one or more bounding boxes surrounding the objects. Specifically, the box generation unit 11 provides image data of an input image to the OD-CNN, and obtains one or more bounding boxes surrounding an object from the OD-CNN. When image data of an input image is provided, the OD-CNN generates one or more bounding boxes surrounding an object in the input image using one or more filters.

[0013] OD-CNN generates feature maps when performing object detection processing. A feature map is a three-dimensional box with dimensions w, h, and c, the number of filters for object detection, in the width direction of the input image and the height direction of the input image, respectively. c=1,...,C. C is an integer equal to or greater than 1. If the types of objects detected by the box generation unit 11 are, for example, humans, dogs, and cats, the OD-CNN performs object detection processing using a filter for human detection, a filter for dog detection, and a filter for cat detection. If the number of these detection filters is one each, then C=3. However, the number of these detection filters is not limited to one each, and the OD-CNN may use multiple filters, for example, as a dog detection filter. If the number of human detection filters is one, the number of dog detection filters is three, and the number of cat detection filters is one, then C=1+3+1=5. Based on the feature map, OD-CNN generates one or more bounding boxes that enclose objects in the input image. If there are multiple objects in the input image, OD-CNN generates one or more bounding boxes for each object. The box generation unit 11 outputs one or more bounding boxes for each object to each of the box selection unit 12 and the overlap calculation unit 13.

[0014] The box selection unit 12 is realized by, for example, a box selection circuit 22 shown in FIG. If the box generation unit 11 has generated a plurality of overlapping bounding boxes for each object, the box selection unit 12 executes a filtering process called NMS. By executing the filtering process, the box selection unit 12 leaves the bounding box with the highest reliability among the multiple bounding boxes and deletes the other bounding boxes. Specifically, the box selection unit 12 calculates the reliability of each bounding box and compares the reliability of the multiple bounding boxes with each other. Based on the result of comparing the reliability, the box selection unit 12 selects the bounding box with the highest reliability from among the multiple bounding boxes. The box selection unit 12 outputs the selected bounding box for each object to both the overlapping degree calculation unit 13 and the display processing unit 15.

[0015] In the object detection device 1 shown in Fig. 1, the box selection unit 12 selects the bounding box with the highest reliability from among a plurality of bounding boxes. However, this is merely an example, and the box selection unit 12 may select, for example, the bounding box with the second highest reliability, as long as there is no practical problem. In the object detection device 1 shown in Fig. 1, the box selection unit 12 selects one bounding box with the highest reliability from among the multiple bounding boxes. However, this is merely an example, and the box selection unit 12 may select the top G bounding boxes with the highest reliability from among the multiple bounding boxes. G is an integer equal to or greater than 2.

[0016] The overlap calculation unit 13 is realized by, for example, an overlap calculation circuit 23 shown in FIG. The overlap calculation unit 13 obtains from the box generation unit 11 one or more bounding boxes for each object. The overlap calculation unit 13 obtains the bounding box selected for each object from the box selection unit 12. The overlap calculation unit 13 calculates the overlap between each bounding box of each object and the selected bounding box.

[0017] Specifically, the overlap calculation unit 13 calculates a first overlap degree, which is an overlap degree between each of the bounding boxes generated by OD-CNN, the number of which corresponds to the number of object detection filters, and the bounding box selected by the box selection unit 12. Then, the overlap calculation unit 13 calculates a second overlap degree, which is a sum of the first overlap degrees, the number of which corresponds to the number of object detection filters. For example, when the type of object is a human, if the number of filters for human detection is three, the overlap calculation unit 13 calculates three first overlap degrees and calculates the sum of the three first overlap degrees as the second overlap degree. For example, when the type of object is a dog, if the number of filters for dog detection is five, the overlap calculation unit 13 calculates five first overlap degrees and calculates the sum of the five first overlap degrees as the second overlap degree. For example, when the type of object is a cat, if the number of filters for cat detection is four, the overlap calculation unit 13 calculates four first overlap degrees and calculates the sum of the four first overlap degrees as the second overlap degree. The overlapping degree calculation section 13 outputs the second overlapping degree to the contribution degree calculation section 14 as the overlapping degree calculated for each object.

[0018] The contribution calculation section 14 is realized by, for example, a contribution calculation circuit 24 shown in FIG. The contribution degree calculation section 14 obtains the second overlap degree for each object from the overlap degree calculation section 13. The contribution degree calculation section 14 calculates the contribution degree of each pixel contributing to the detection of the object among the plurality of pixels included in the input image, based on the second overlap degree for each object. The contribution degree calculation unit 14 outputs to the display processing unit 15 an image indicating the contribution degree of the pixels contributing to the detection of each object.

[0019] The display processing unit 15 is realized by, for example, a display processing circuit 25 shown in FIG. The display processing unit 15 obtains from the box selection unit 12 the bounding box selected for each object. Furthermore, the display processing unit 15 acquires from the contribution calculation unit 14 an image indicating the contribution of pixels contributing to the detection of each object. The display processing unit 15 causes the bounding box selected by the box selection unit 12 to be displayed on the display of the display device 2. Furthermore, the display processing unit 15 causes the display device 2 to display an image indicating the degree of contribution of pixels contributing to the detection of each object.

[0020] 1, it is assumed that each of the components of the object detection device 1, that is, the box generation unit 11, the box selection unit 12, the overlapping degree calculation unit 13, the contribution degree calculation unit 14, and the display processing unit 15, is realized by dedicated hardware as shown in Fig. 2. In other words, it is assumed that the object detection device 1 is realized by a box generation circuit 21, a box selection circuit 22, an overlapping degree calculation circuit 23, a contribution degree calculation circuit 24, and a display processing circuit 25. Each of the box generation circuit 21, the box selection circuit 22, the overlap calculation circuit 23, the contribution calculation circuit 24 and the display processing circuit 25 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of these.

[0021] The components of the object detection device 1 are not limited to those realized by dedicated hardware, and the object detection device 1 may be realized by software, firmware, or a combination of software and firmware. The software or firmware is stored as a program in the memory of a computer. The computer means hardware that executes the program, and includes, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a central processing unit, a processing unit, an arithmetic unit, a microprocessor, a microcomputer, a processor, or a DSP (Digital Signal Processor).

[0022] FIG. 3 is a hardware configuration diagram of a computer in the case where object detection device 1 is realized by software, firmware, or the like. When the object detection device 1 is realized by software, firmware, or the like, a program for causing a computer to execute the respective processing procedures of the box generation unit 11, the box selection unit 12, the overlap calculation unit 13, the contribution calculation unit 14, and the display processing unit 15 is stored in the memory 31. Then, a processor 32 of the computer executes the program stored in the memory 31.

[0023] 2 shows an example in which each of the components of the object detection device 1 is realized by dedicated hardware, and Fig. 3 shows an example in which the object detection device 1 is realized by software, firmware, etc. However, this is merely one example, and some of the components in the object detection device 1 may be realized by dedicated hardware, and the remaining components may be realized by software, firmware, etc.

[0024] Next, the operation of the object detection device 1 shown in FIG. 1 will be described. FIG. 4 is a flowchart showing an object detection method which is a processing procedure of the object detection device 1. The box generating unit 11 acquires image data of an input image, for example, as shown in FIG. 5, from the outside. FIG. 5 is an explanatory diagram illustrating an example of an input image. In the example of FIG. 5, a human and a dog are shown as objects in the input image.

[0025] The box generation unit 11 provides image data of an input image to the OD-CNN. When image data of an input image is provided from the box generation unit 11, the OD-CNN uses one or more object detection filters to generate one or more bounding boxes X surrounding an object in the input image. i,j (Step ST1 in FIG. 4). If multiple objects are captured in the input image, the box generating unit 11 generates one or more bounding boxes X i,1 ,···,X i,J Generate.

[0026] FIG. 6 is an explanatory diagram showing an example of bounding box generation by the box generating unit 11. As shown in FIG. i=1,...,I, where I is the number of objects in the input image. For example, if the number of objects in the input image is two, I=2, and if the number of objects in the input image is three, I=3. In the example of Figure 5, the number of objects in the input image is two. For each object, one or more bounding boxes X i,1 ,···,X i,J are overlapping each other. j=1,...,J. J is the number of object detection filters for each object, and is the number of bounding boxes generated for each object. For example, when the object type is a human, if the number of human detection filters is three, J=3. For example, when the object type is a dog, if the number of dog detection filters is five, J=5. For example, when the object type is a cat, if the number of cat detection filters is four, J=4. The process of generating a bounding box is a known technique, and therefore a detailed description thereof will be omitted.

[0027] In the example of FIG. 6, the number of filters for human detection and the number of filters for dog detection are 3, and the three bounding boxes X 1,1 ,X 1,2 ,X 1,3 and the three bounding boxes X that surround the dog. 2,1 ,X 2,2 ,X 2,3 , and are generated. Bounding Box X i,j is a rectangular area. Bounding Box X i,j is the attribute value, and the class probability X i,j [cp,1]···X i,j [cp,M] and the coordinates X of the four vertices of the rectangular area i,j [w1,h1],X i,j [w2,h2],X i,j [w3,h3],X i,j [w4,h4].

[0028] Class Probability X i,j [cp,m](m=1, ,M) is the bounding box X i,j is the probability that the type of object surrounded by is a certain type of object. Specifically, the class probability X i,j [cp,1] is the bounding box X i,j The probability that the object surrounded by is a human, class probability X i,j [cp,2] is the bounding box X i,j The probability that the object surrounded by is a dog, class probability X i,j [cp,3] is the bounding box X i,j The probability that the object surrounded by is a cat is shown. M is an integer greater than or equal to 1. X i,j [w1,h1] is the coordinate of the upper left vertex of the four vertices of the rectangular area. X i,j [w2,h2] is the coordinate of the upper right corner of the four vertices of the rectangular area. X i,j [w3,h3] is the coordinate of the lower left vertex of the four vertices of the rectangular area. Xi,j [w4,h4] are the coordinates of the lower right vertex of the four vertices of the rectangular area. The box generator 11 generates one or more bounding boxes X for each object. i,j to the box selection unit 12 and the overlap calculation unit 13.

[0029] The box selection unit 12 receives one or more bounding boxes X for each object from the box generation unit 11. i,j Get the. The box selection unit 12 selects a bounding box X for each object. i,j If the number is 1, the bounding box X i,j the bounding box of the selected X i ' to the overlap degree calculation unit 13 and the display processing unit 15, respectively. The box selection unit 12 selects a bounding box X for each object. i,j If there are multiple objects, multiple bounding boxes X i,j A filtering process called NMS is performed on the received data. The box selection unit 12 performs a filtering process to select a plurality of bounding boxes X i,j Among them, the bounding box with the highest confidence is kept and the other bounding boxes are deleted.

[0030] Specifically, the box selection unit 12 selects a bounding box X i,j Reliability of DR i,j Calculate the reliability DR. i,j The calculation process itself is a known technique, and therefore a detailed description thereof will be omitted. Bounding Box X i,j For example, if the bounding box surrounding a human is X i,j Reliability DR i,j As a bounding box X i,j Class probability of Xi,j [cp,1]···X i,j Among [cp,M], the class probability that the surrounding object is a human can be used. Bounding Box X i,j But for example, if you want to define a bounding box around a dog, you would use bounding box X i,j Reliability DR i,j As a bounding box X i,j Class probability of X i,j [cp,1]···X i,j Among [cp,M], we can use the class probability that the surrounding object is a dog. The box selection unit 12 selects each bounding box X i,j Reliability DR i,j Based on multiple bounding boxes X i,1 ,···,X i,J Among them, the bounding box X with the highest confidence is i ' is selected (step ST2 in FIG. 4). The box selection unit 12 selects a bounding box X for each object. i ' to the overlap degree calculation unit 13 and the display processing unit 15, respectively.

[0031] The overlap calculation unit 13 receives one or more bounding boxes X for each object from the box generation unit 11. i,j Get the. The overlap calculation unit 13 also receives the bounding box X selected for each object from the box selection unit 12. i ' to get '. The overlap calculation unit 13 calculates the bounding box X i,j and the selected bounding box X i ' and overlap DS i,j (Step ST3 in FIG. 4). The overlap degree calculation unit 13 calculates the overlap degree DS for each object. i,j to the contribution calculation unit 14.

[0032] Hereinafter, the process of calculating the degree of overlap by the overlap degree calculation unit 13 will be specifically described. Figure 7 shows the bounding box X i,j and bounding box X i ' and overlap DS i,j FIG. The overlap calculation unit 13 calculates the bounding box X i,j The coordinates of the four vertices are the attribute values ​​of i,j [w1,h1],X i,j [w2,h2],X i,j [w3,h3],X i,j Based on [w4,h4], we find the bounding box X i,j Area W i,j Calculate. The overlap calculation unit 13 calculates the bounding box X i ' The coordinates of the four vertices are the attribute values ​​of i '[w1,h1],X i '[w2,h2],X i '[w3,h3],X i 'Based on [w4,h4], we find the bounding box X i Area of ​​W' i ' is calculated.

[0033] Next, the overlap calculation unit 13 calculates the bounding box X i,j The coordinates of the four vertices, which are the attribute values ​​of the bounding box X i ' attribute values ​​and the coordinates of the four vertices, and the bounding box X i,j and bounding box X i The coordinates of the four vertices of the overlapping area of ​​OA i,j [w1,h1],OA i,j [w2,h2],OA i,j [w3,h3],OA i,j Calculate [w4,h4]. In the example of Figure 7, the overlapping areas are the shaded areas. The overlap degree calculation unit 13 calculates the coordinates OA of the four vertices of the overlapping areas. i,j [w1,h1],OA i,j [w2,h2],OAi,j [w3,h3],OA i,j Based on [w4,h4], the area of ​​the overlapping regions (X i,j ∩X i ') is calculated.

[0034] Next, the overlap calculation unit 13 calculates the bounding box X i,j Area W i,j and bounding box X i Area of ​​W' i ', the area of ​​the overlapping regions (X i,j ∩X i ') to get the bounding box X i,j and bounding box X i ' and the area of ​​the union (X i,j ∪X i ') is calculated. The overlap calculation unit 13 calculates the area (X i,j ∩X i ') to the area of ​​the union (X i,j ∪X i ') to find the bounding box X i,j and bounding box X i 'The first overlap with DS(1) i,j Calculate. DS(1) i,j =(X i,j ∩X i ') / (X i,j ∪X i ') (1)

[0035] The overlap calculation unit 13 calculates J bounding boxes X i,1 ,···,X i,J For each of the first overlap DS(1) i,1 ,···,DS(1) i,J When the first overlap degree DS(1) is calculated, the J first overlap degrees DS(1) are calculated as shown in the following formula (2). i,1 ,···,DS(1) i,J The second overlap DS(2) is calculated by calculating the sum of i Calculate. DS(2) i =DS(1) i,1 +···+DS(1) i,J (2) The overlap degree calculation unit 13 calculates a second overlap degree DS(2) for each object. i The overlapping degree DS for each object i,j to the contribution calculation unit 14.

[0036] The contribution calculation unit 14 receives the overlapping degree DS for each object from the overlapping degree calculation unit 13. i,j Get the. The contribution calculation unit 14 calculates the overlapping degree DS for each object using pixels included in the input image. i,j By calculating the partial differential of , the contribution of pixels that contribute to the detection of an object among a plurality of pixels included in the input image is calculated (step ST4 in FIG. 4). The contribution degree calculation section 14 outputs to the display processing section 15 an image indicating the contribution degree of the pixels contributing to the detection of each object.

[0037] The process of calculating the contribution degree by the contribution degree calculation section 14 will be specifically described below. The contribution degree calculation process can use, for example, a technique called Gradient-weighted Class Activation Mapping (Grad-CAM) described below. The contribution calculation unit 14 calculates the bounding box X i 'Overlap DS i,j By backpropagating the error, we obtain the bounding box X i Calculate the gradient of the bounding box X for the convolutional layer. i The process of calculating the gradient of ' is a known technique, and therefore a detailed description thereof will be omitted. Next, the contribution calculation unit 14 obtains a global average pooling (hereinafter referred to as "GAP") of the gradients of each convolution layer, and performs weighted addition of the multiple convolution layers using each GAP as a weighting coefficient. The contribution calculation unit 14 generates a Grad-CAM image as shown in FIG. 8 by resizing the image indicating the weighted addition result of a plurality of convolution layers to the image size of the input image.

[0038] FIG. 8 is an explanatory diagram showing an example of an image of Grad-CAM. A Grad-CAM image is an image that shows the contribution of pixels to object detection, with the color of the pixels being changed according to their contribution. In the example of Figure 8, the pixel with the highest contribution is displayed in pink, the pixel with the second highest contribution is displayed in yellow, and the pixel with the third highest contribution is displayed in green, although this is by way of example only and the contributions may be displayed in different colors.

[0039] The display processing unit 15 obtains from the box selection unit 12 the bounding box selected for each object. The display processing unit 15 displays the bounding box selected by the box selection unit 12 on the display of the display device 2 (see FIG. 6). Moreover, the display processing unit 15 acquires a Grad-CAM image indicating the contribution of pixels for each object from the contribution calculation unit 14. The display processing unit 15 acquires, for example, a Grad-CAM image as shown in FIG. The display processing unit 15 causes the Grad-CAM image to be displayed on the display of the display device 2 (step ST5 in FIG. 4).

[0040] In the above-described first embodiment, the object detection device 1 is configured to include a box generation unit 11 that detects an object in an input image and generates a bounding box surrounding the object, and a box selection unit 12 that selects one of the bounding boxes based on the reliability of each bounding box if multiple bounding boxes that overlap each other are generated by the box generation unit 11. The object detection device 1 also includes an overlap calculation unit 13 that calculates the overlap between each of the bounding boxes generated by the box generation unit 11 and the bounding box selected by the box selection unit 12, and a contribution calculation unit 14 that calculates the contribution of pixels that contribute to the detection of the object among the multiple pixels included in the input image based on the overlap calculated by the overlap calculation unit 13. Thus, the object detection device 1 can present the contribution of pixels that contribute to the detection of the object among the multiple pixels included in the input image.

[0041] Embodiment 2 In the second embodiment, the overlap calculation unit 17 calculates J bounding boxes X i,1 ,···,X i,J The first overlap DS(1) for each i,1 ,···,DS(1) i,J Next, an object detection device 1 that performs weighted addition will be described.

[0042] Fig. 9 is a configuration diagram showing an object detection device 1 according to embodiment 2. In Fig. 9, the same reference numerals as in Fig. 1 indicate the same or corresponding parts, and detailed description thereof will be omitted. Fig. 10 is a hardware configuration diagram showing the hardware of object detection device 1 according to embodiment 2. In Fig. 10, the same reference numerals as in Fig. 2 denote the same or corresponding parts, and detailed description thereof will be omitted. The object detection device 1 shown in FIG. 9 includes a box generation unit 11, a box selection unit 12, a weighting reception unit 16, an overlapping degree calculation unit 17, a contribution degree calculation unit 14, and a display processing unit 15.

[0043] The weighting reception unit 16 is realized by, for example, a weighting reception circuit 26 shown in FIG. The weighting reception unit 16 receives the first overlapping degree DS(1) i,1 ,···,DS(1) i,J Weighting factor wf i,1 ,···,wf i,J Get the. The weighting reception unit 16 receives the weighting coefficient wf i,1 ,···,wf i,J to the overlap degree calculation unit 17.

[0044] The overlap calculation unit 17 is realized by, for example, an overlap calculation circuit 27 shown in FIG. The overlap calculation unit 17 receives one or more bounding boxes X for each object from the box generation unit 11. i,j Get the. The overlap calculation unit 17 receives the bounding box X selected for each object from the box selection unit 12. i ' to get '. The overlap calculation unit 17 receives the weighting coefficient wf i,j Get the. The overlap calculation unit 17 calculates the bounding box X i,j and bounding box X i The first overlap degree DS(1) is the overlap degree with i,j Calculate. The overlap degree calculation unit 17 calculates a first overlap degree DS(1) i,j For the weighting factor wf i,j The first overlap degree DS(1) after multiplication by J weight coefficients is obtained. i,j ×wf i,j The second overlap DS(2) is obtained by adding i Calculate. The overlapping degree calculation unit 17 calculates a second overlapping degree DS(2) for each object. i The overlapping degree DS for each object i and outputs the result to the contribution calculation unit 14.

[0045] 9, it is assumed that each of the components of the object detection device 1, namely, the box generation unit 11, the box selection unit 12, the weighting reception unit 16, the overlapping degree calculation unit 17, the contribution calculation unit 14, and the display processing unit 15, is realized by dedicated hardware as shown in Fig. 10. That is, it is assumed that the object detection device 1 is realized by a box generation circuit 21, a box selection circuit 22, a weighting reception circuit 26, an overlapping degree calculation circuit 27, a contribution calculation circuit 24, and a display processing circuit 25. Each of the box generation circuit 21, the box selection circuit 22, the weighting reception circuit 26, the overlap calculation circuit 27, the contribution calculation circuit 24 and the display processing circuit 25 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination of these.

[0046] The components of the object detection device 1 are not limited to those realized by dedicated hardware, and the object detection device 1 may be realized by software, firmware, or a combination of software and firmware. When object detection device 1 is realized by software, firmware, or the like, programs for causing a computer to execute the respective processing procedures of box generation unit 11, box selection unit 12, weighting reception unit 16, overlapping degree calculation unit 17, contribution degree calculation unit 14, and display processing unit 15 are stored in memory 31 shown in Fig. 3. Then, processor 32 shown in Fig. 3 executes the programs stored in memory 31.

[0047] 10 shows an example in which each of the components of the object detection device 1 is realized by dedicated hardware, and Fig. 3 shows an example in which the object detection device 1 is realized by software, firmware, etc. However, this is merely one example, and some of the components in the object detection device 1 may be realized by dedicated hardware, and the remaining components may be realized by software, firmware, etc.

[0048] Next, the operation of the object detection device 1 shown in Fig. 9 will be described. Other than the weighting reception unit 16 and the overlapping degree calculation unit 17, the object detection device 1 is the same as that shown in Fig. 1. Therefore, here, the operation of the weighting reception unit 16 and the overlapping degree calculation unit 17 will mainly be described.

[0049] The user, for example, operates a man-machine interface (not shown) to input the first overlap degree DS(1) i,1 ,···,DS(1) i,J Weighting factor wf i,1 ,···,wf i,J Set. The man-machine interface (not shown) is a weighting factor wf i,1 ,···,wf i,J to the weighting receiving unit 16. In the object detection device 1 shown in FIG. 9, a man-machine interface (not shown) determines the weighting coefficient wf i,1 ,···,wf i,J to the weighting reception unit 16. However, this is merely an example, and an external device of the object detection device 1 may output the weighting coefficient wf i,1 ,···,wf i,J may be transmitted to the weighting receiving unit 16.

[0050] The weighting reception unit 16 receives the first overlapping degree DS(1) from a man-machine interface (not shown). i,1 ,···,DS(1) i,J Weighting factor wf i,1 ,···,wf i,J Get the. The weighting reception unit 16 receives the weighting coefficient wf i,1 ,···,wf i,J to the overlap degree calculation unit 17.

[0051] The overlap calculation unit 17 receives one or more bounding boxes X for each object from the box generation unit 11. i,j Get the. The overlap calculation unit 17 receives the bounding box X selected for each object from the box selection unit 12. i' to get '. The overlap calculation unit 17 receives the weighting coefficient wf i,j Get the. The overlap calculation unit 17, like the overlap calculation unit 13 shown in FIG. i,j and bounding box X i The first overlap degree DS(1) is the overlap degree with i,j Calculate. The overlap degree calculation unit 17 calculates the first overlap degree DS(1) as shown in the following equation (3). i,j For the weighting factor wf i,j The first overlap degree DS(1) after multiplication by J weight coefficients is obtained. i,j ×wf i,j The second overlap DS(2) is obtained by adding i Calculate. DS(2) i =DS(1) i,1 ×wf i,1 +···+DS(1) i,J ×wf i,J (3) The overlapping degree calculation unit 17 calculates a second overlapping degree DS(2) for each object. i The overlapping degree DS for each object i and outputs the result to the contribution calculation unit 14.

[0052] In the above-described second embodiment, the object detection device 1 is configured so that the overlap calculation unit 17 calculates the second overlap by weighting and adding the first overlaps for the number of filters. Therefore, the object detection device 1 can present the contribution of pixels that contribute to object detection among a plurality of pixels included in an input image, and can present the contribution of a pixel by adding a weight to a specific filter among one or more filters used by OD-CNN.

[0053] Embodiment 3 In the third embodiment, an object detection device 1 in which a plurality of sets of box generating units 11 and box selecting units 12 are connected in cascade will be described. That is, in the third embodiment, an object detection device 1 in which a plurality of pairs of OD-CNN and NMS are connected in cascade will be described.

[0054] Fig. 11 is a configuration diagram showing an object detection device 1 according to embodiment 3. In Fig. 11, the same reference numerals as in Fig. 1 and Fig. 9 indicate the same or corresponding parts, and detailed description thereof will be omitted. 11, N pairs of box generating units 11 and box selecting units 12 are cascade-connected. N is an integer of 2 or more. The set 41-n (n=1, . . . , N) includes a box generating unit 11 and a box selecting unit 12, as well as an overlapping degree calculating unit 13 and a contribution degree calculating unit . The set 41-1 is a set in the first row, and an input image is provided to the box generation unit 11 of the set 41-1. The box selection unit 12 of the set 41-1 outputs the selected bounding box to each of the box generation unit 11 and the display processing unit 15 of the set in the second row. The contribution calculation section 14 of the group 41-1 outputs the calculated contribution of the pixel to the display processing section 15.

[0055] The box generation unit 11 of the set 41-n (n=2, . . . , N) obtains the bounding box selected by the box selection unit 12 of the set 41-(n-1) and outputs the bounding box to the box selection unit 12 of the same set. The box selection unit 12 of a set 41-n (n=2, . . . , N) outputs the selected bounding box to each of the overlap calculation unit 13 of the same set, and the box generation unit 11 and display processing unit 15 of the set 41-(n+1). If the set 41-n is the final-stage set 41-N, the box selection unit 12 outputs the selected bounding box to each of the overlap calculation unit 13 and display processing unit 15 of the same set. The contribution calculation section 14 of the set 41-n (n=2, . . . , N) outputs the calculated contribution of the pixel to the display processing section 15.

[0056] In the object detection device 1 shown in Fig. 11, a set 41-n (n = 1,...,N) includes the box generation unit 11, the box selection unit 12, the overlap calculation unit 13, and the contribution calculation unit 14, which are included in the object detection device 1 shown in Fig. 1. However, this is merely an example, and the set 41-n (n = 1,...,N) may include the box generation unit 11, the box selection unit 12, the weighting reception unit 16, the overlap calculation unit 17, and the contribution calculation unit 14, which are included in the object detection device 1 shown in Fig. 9.

[0057] Next, the operation of the object detection device 1 shown in FIG. 11 will be described. The operations of the box generating unit 11, the box selecting unit 12, the overlapping degree calculating unit 13 and the contribution degree calculating unit 14 are similar to those of the object detection device 1 shown in FIG. The overlap calculation unit 13 for set 41-n (n = 1, ..., N) calculates the overlap between each bounding box generated by the box generation unit 11 for set 41-n and the bounding box selected by the box selection unit 12 for set 41-n.

[0058] The contribution calculation unit 14 of the set 41-1 calculates the contribution of pixels that contribute to the detection of an object among the multiple pixels included in the input image based on the overlap calculated by the overlap calculation unit 13 of the set 41-1. The contribution calculation unit 14 of the group 41-1 outputs to the display processing unit 15 an image of the Grad-CAM indicating the contribution of the pixel. The contribution calculation unit 14 of the set 41-n (n = 2, ..., N) calculates the contribution of pixels contributing to the detection of an object among the multiple pixels included in the bounding box selected by the box selection unit 12 of the set 41-(n-1) based on the overlap calculated by the overlap calculation unit 13 of the set 41-n. The contribution calculation unit 14 for the set 41-n outputs to the display processing unit 15 an image of the Grad-CAM indicating the contribution of the pixel.

[0059] The display processing unit 15 acquires the Grad-CAM image from the contribution calculation unit 14 of the set 41-n (n=1, . . . , N), and causes the display device 2 to display the Grad-CAM image on its display.

[0060] In the above-described third embodiment, a plurality of pairs 41-n (n=1, . . . , N) of a box generating unit 11 and a box selecting unit 12 are connected in cascade. The object detection device 1 is configured such that each of the cascade-connected pairs 41-n includes an overlapping degree calculation unit 13 and a contribution degree calculation unit 14. Therefore, even if a plurality of pairs of an OD-CNN and an NMS are connected in cascade, the object detection device 1 can present the contribution degree of a pixel in each pair.

[0061] In addition, the present disclosure allows free combination of the respective embodiments, modification of any of the components of each embodiment, or omission of any of the components of each embodiment. [Industrial Applicability]

[0062] The present disclosure is suitable for an object detection device and an object detection method. [Explanation of symbols]

[0063] 1 object detection device, 2 display device, 11 box generation unit, 12 box selection unit, 13 overlap calculation unit, 14 contribution calculation unit, 15 display processing unit, 16 weighting reception unit, 17 overlap calculation unit, 21 box generation circuit, 22 box selection circuit, 23 overlap calculation circuit, 24 contribution calculation circuit, 25 display processing circuit, 26 weighting reception circuit, 27 overlap calculation circuit, 31 memory, 32 processor, 41-1 to 41-N sets.

Claims

1. a box generator for detecting objects in an input image and generating one or more bounding boxes enclosing the objects; a box selection unit that, if a plurality of bounding boxes overlapping each other are generated by the box generation unit, selects one of the plurality of bounding boxes based on a reliability of each of the bounding boxes; an overlap calculation unit that calculates an overlap between each of the bounding boxes generated by the box generation unit and the bounding box selected by the box selection unit; a contribution degree calculation unit that calculates a contribution degree of a pixel that contributes to the detection of the object among a plurality of pixels included in the input image based on the overlap degree calculated by the overlap degree calculation unit; An object detection device comprising:

2. The box generation unit Detecting objects in the input image using a convolutional neural network object detection algorithm; The convolutional neural network includes:

2. The object detection apparatus according to claim 1, wherein, given the input image, one or more filters are each used to generate a bounding box surrounding an object in the input image.

3. The overlap degree calculation unit Calculating a first overlap degree, which is an overlap degree between each of the bounding boxes generated by the convolutional neural network, the number of which corresponds to the number of the filters, and the bounding box selected by the box selection unit, and calculating a second overlap degree, which is a sum of the first overlap degrees for the number of the filters; The contribution calculation unit is 3. The object detection device according to claim 2, further comprising: a calculation unit for calculating a contribution degree of pixels that contribute to the detection of the object, among a plurality of pixels included in the input image, based on the second overlap degree calculated by the overlap degree calculation unit.

4. The overlap degree calculation unit 4. The object detection device according to claim 3, wherein the second overlapping degree is calculated by weighting and adding the first overlapping degrees for the number of the filters.

5. The object detection device according to claim 1 , further comprising a display processing unit that displays an image indicating the contribution degree calculated by the contribution degree calculation unit.

6. a plurality of pairs of the box generating unit and the box selecting unit are connected in cascade; Each pair connected in cascade is The object detection device according to claim 1 , further comprising: the overlap calculation unit and the contribution calculation unit.

7. a box generator for detecting objects in the input image and generating one or more bounding boxes surrounding the objects; a box selection unit, if a plurality of bounding boxes overlapping each other are generated by the box generation unit, selecting one of the plurality of bounding boxes based on a reliability of each of the bounding boxes; an overlap calculation unit calculates an overlap between each of the bounding boxes generated by the box generation unit and the bounding box selected by the box selection unit; A contribution degree calculation unit calculates a contribution degree of a pixel that contributes to the detection of the object among a plurality of pixels included in the input image based on the overlap degree calculated by the overlap degree calculation unit. Object detection methods.

Images