Information processing system, information processing method, and program

Abstract

To provide an object detection device that efficiently and easily selects images for creating teacher data on the basis of the number of detected objects.SOLUTION: An object detection device includes: a detection part that uses a dictionary to detect an object from each of a plurality of input images; a reception part that displays a graph showing a relationship between the input images and the number of subregions where the object has been detected on a display device, and displays one of the plurality input images according to the position on the graph accepted by an operation of an input device on the display device for creation of teacher data; a generation part that generates teacher data from the input images; and a learning part that uses the teacher data to learn the dictionary.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an object detection device that detects an object from an image using a dictionary. [Background technology]

[0002] Devices that analyze images captured by an imaging device and detect objects have been proposed or put into practical use. For example, Patent Document 1 discloses a technique for analyzing images captured by an imaging device during a manufacturing process of a board such as a printed wiring board, and detecting objects such as scratches on the board. Patent Document 2 discloses a technique for analyzing monitor images of a road captured by a surveillance camera and detecting objects such as vehicles.

[0003] To perform the above-mentioned detection, the object detection device needs to learn to analyze images and videos. And for this learning, teacher data (also called training data) is required. Teacher data is examples of pairs of input and output. There are two types of teacher data: positive examples and negative examples. To properly train an object detection device, both positive and negative examples are necessary. However, creating appropriate teacher data requires a lot of time and effort. For this reason, several technologies have been proposed to assist in creating teacher data.

[0004] For example, the above-mentioned Patent Document 1 (see FIG. 9) discloses a method for creating teacher data required for detecting objects such as scratches on a board. In this method, areas having brightness values ​​different from those of non-defective products are extracted from an image of a printed wiring board and displayed on a display, and the selection of an area and input of its class (also called category) are accepted from a user using a keyboard and mouse. Specifically, the user selects one specific area from among multiple existing areas by clicking the mouse, and then selects the desired class from a pull-down menu that is displayed at the time of selection by clicking the mouse.

[0005] Furthermore, the above-mentioned Patent Document 2 discloses a method for creating training data required for detecting objects such as vehicles traveling on a road. This method automatically performs a series of operations: cutting out an object area from an arbitrary captured image using a dictionary, extracting a predetermined feature amount from the area, and learning the above-mentioned dictionary based on the extracted feature amount. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2003-317082 A [Patent Document 2] JP 2014-59729 A Summary of the Invention [Problem to be solved by the invention]

[0007] There may be many images in which object regions are detected using a dictionary from images captured by a camera. In this case, users may wish to select the order of images to be checked for creating training data based on the number of detected objects. Currently, in order to check the number of objects detected in an image, each image must be displayed on a display device and checked. However, this method requires a lot of time and effort to select images to be processed based on the number of detected objects.

[0008] In view of the above-mentioned problems, an object of the present invention is to provide an object detection device and the like that efficiently and easily selects images for creating training data based on the number of detected objects. [Means for solving the problem]

[0009] The first feature of the present invention is a detection unit that detects an object from each of a plurality of input images using the dictionary; a reception unit that displays, on a display device, a graph indicating a relationship between the input image and the number of partial regions in which the object is detected, and displays, on the display device, one of the plurality of input images in accordance with a position on the graph received by an operation of the input device, for generating teacher data; A generation unit that generates training data from the input image; a learning unit that learns the dictionary using the teacher data; The object detection apparatus has the following features.

[0010] The second feature of the present invention is Detecting an object in each of a plurality of input images using the dictionary; displaying on a display device a graph showing a relationship between the input image and the number of partial regions in which the object is detected; displaying one of the plurality of input images on a display device in accordance with a position on the graph received by operating an input device for generating training data; generating training data from the input image; The object detection method learns the dictionary using the training data.

[0011] The third feature of the present invention is Computer, a detection unit that detects an object from each of a plurality of input images using the dictionary; a reception unit that displays, on a display device, a graph indicating a relationship between the input image and the number of partial regions in which the object is detected, and displays, on the display device, one of the plurality of input images in accordance with a position on the graph received by an operation of the input device, for generating teacher data; A generation unit that generates training data from the input image; a learning unit that learns the dictionary using the teacher data; It is a recording medium that stores a program that causes the device to function as described above. Effect of the Invention

[0012] The present invention can efficiently and easily select images for creating training data based on the number of detected objects. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram of an object detection device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is an explanatory diagram of the concept of detecting an object from an image in the first embodiment of the present invention. [Diagram 3] FIG. 4 is a diagram showing an example of an object detection result in the first embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing an example of a reception screen in the first embodiment of the present invention. [Diagram 5] 5 is a flowchart showing an example of the operation of the object detection device according to the first embodiment of the present invention. [Figure 6] 5 is a flowchart showing details of the operation of a reception unit of the object detection device according to the first embodiment of the present invention. [Figure 7] FIG. 13 is a diagram showing an example of a combination of gesture types that accept an input of class 0 and a combination of gesture types that accept an input of an increment of a class in a modified example of the first embodiment of the present invention. [Figure 8] FIG. 13 is a diagram showing an example of combinations of gesture types that accept inputs of class 0, class 1, and class 2 in a modified example of the first embodiment of the present invention. [Figure 9] FIG. 11 is a diagram showing an example of a reception screen in the second embodiment of the present invention. [Figure 10A] 13A and 13B are diagrams illustrating the meaning of drag-and-drop in the lower right direction used in the second embodiment of the present invention and an example of calculating a partial region. [Figure 10B] 13A and 13B are diagrams illustrating the meaning of drag-and-drop in the lower left direction used in the second embodiment of the present invention and an example of calculating a partial region. [Figure 11] 10 is a flowchart showing details of the operation of a reception unit of an object detection device according to a second embodiment of the present invention. [Figure 12A] FIG. 11 is a diagram showing an example of a detection result before being updated by a reception unit of an object detection device according to a second embodiment of the present invention. [Figure 12B] FIG. 11 is a diagram showing an example of a detection result after being updated by a receiving unit of an object detection device according to a second embodiment of the present invention. [Figure 13A] FIG. 11 is a diagram showing an example of an image window before it is updated by a reception unit of an object detection device according to a second embodiment of the present invention. [Figure 13B] FIG. 11 is a diagram showing an example of an image window after being updated by a reception unit of an object detection device according to a second embodiment of the present invention. [Figure 14] FIG. 13 is a diagram showing an example of a combination of gesture types in a modified example of the second embodiment of the present invention. [Figure 15] FIG. 13 is an explanatory diagram of a method for calculating a partial area of ​​class 2 from the position of a right double click performed on an image in a modified example of the second embodiment of the present invention. [Figure 16A] 13A and 13B are diagrams illustrating the meaning of a swipe in the lower right direction and an example of calculating a partial region used in a modified example of the second embodiment of the present invention. [Figure 16B] 13A and 13B are diagrams illustrating an example of calculation of a partial region and the meaning of a swipe in the lower left direction used in a modified example of the second embodiment of the present invention. [Figure 17A] 13A and 13B are diagrams illustrating an example of calculation of a partial region and the meaning of a downward swipe used in a modified example of the second embodiment of the present invention. [Figure 17B] 13A and 13B are diagrams illustrating an example of calculation of a partial region and the meaning of an upward swipe used in a modified example of the second embodiment of the present invention. [Figure 18] FIG. 13 is a diagram showing an example of a combination of gesture types in a modified example of the second embodiment of the present invention. [Figure 19] 13 is a flowchart showing details of the operation of a reception unit of an object detection device according to a third embodiment of the present invention. [Figure 20] FIG. 13 is a diagram showing an example of a reception screen used in the fourth embodiment of the present invention. [Figure 21]FIG. 13 is a diagram showing an example in which a receiving unit of an object detection device according to a fourth embodiment of the present invention receives an input of class 0 for an existing partial region that partially overlaps with a newly calculated partial region. [Figure 22] 13 is a flowchart showing details of the operation of a reception unit of an object detection device according to a fourth embodiment of the present invention. [Figure 23A] FIG. 13 is a diagram showing an example of a detection result before being updated by a reception unit of an object detection device according to a fourth embodiment of the present invention. [Figure 23B] FIG. 13 is a diagram showing an example of a detection result after being updated by a receiving unit of an object detection device according to a fourth embodiment of the present invention. [Figure 24A] FIG. 13 is a diagram showing an example of an image window before it is updated by a reception unit of an object detection device according to a fourth embodiment of the present invention. [Figure 24B] FIG. 13 is a diagram showing an example of an image window after being updated by a reception unit of an object detection device according to a fourth embodiment of the present invention. [Diagram 25] FIG. 13 is a block diagram of an object detection device according to a fifth embodiment of the present invention. [Figure 26] 1 is a diagram illustrating an example of a hardware configuration of an object detection device according to a first embodiment of the present invention. [Figure 27] FIG. 13 is a block diagram of an object detection device according to a sixth embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Next, an embodiment of the present invention will be described in detail with reference to the drawings. In the following description of the drawings, the same or similar parts are given the same or similar reference numerals. However, the drawings are schematic representations of the configuration of the embodiment of the present invention. Furthermore, the embodiment of the present invention described below is merely an example, and can be modified as appropriate within the scope of the same essence. [First embodiment] 1, an object detection device 100 according to a first embodiment of the present invention analyzes a monitor video of a road captured by a surveillance camera 110 to detect an object. In this embodiment, there are two types of objects to be detected: a two-wheeled vehicle and a four-wheeled vehicle. That is, the object detection device 100 is a type of multi-class classifier, and detects two specific types of objects from an image.

[0015] The object detection device 100 has, as main functional units, a detection unit 101, a dictionary 102, a detection result DB (detection result database) 103, a reception unit 104, a generation unit 105, a teacher data memory 106, and a learning unit 107. A display device 120 and an input device 130 are connected to the object detection device 100. For example, as shown in FIG. 26, the object detection device 100 may be configured by an information processing device 230 and a storage medium for recording a program 240. The information processing device 230 has an arithmetic processing unit 210 such as one or more microprocessors, and a storage unit 220 such as a semiconductor memory or a hard disk. The storage unit 220 stores the dictionary 102, the detection result DB 103, the teacher data memory 106, and the like. The program 240 is loaded into memory from an external computer-readable recording medium when the object detection device 100 is started up, etc., and by controlling the operation of the calculation processing unit 210, it realizes functional means such as a detection unit 101, a reception unit 104, a generation unit 105, and a learning unit 107 on the calculation processing unit 210.

[0016] The display device 120 is a screen display device such as an LCD (Liquid Crystal Display) or a PDP (Plasma Display Panel), and displays various information such as detection results on the screen in response to instructions from the object detection apparatus 100.

[0017] The input device 130 is an operation input device such as a keyboard or a mouse. The input device 130 detects an operator's operation as an input and outputs it to the object detection device 100. In this embodiment, a mouse is used as the input device 130. The mouse used is capable of performing two types of mouse gestures, a left click and a right click.

[0018] The detection unit 101 inputs images captured by the surveillance camera 110 one by one in chronological order, and detects objects from each input image using the dictionary 102. The detection unit 101 stores the detection results in the detection result DB 103. One image refers to one frame of an image captured by the surveillance camera 110. Consecutive frame numbers are assigned to the series of frames output from the surveillance camera 110 for identification purposes.

[0019] FIG. 2 is an explanatory diagram of the concept of detecting an object from an image. The detection unit 101 sets a search window 112 for object detection in the image 111, and extracts a feature amount of the image in the search window 112. Furthermore, the detection unit 101 calculates a likelihood indicating a similarity between the extracted feature amount and the dictionary 102, and detects the object by determining whether the image in the search window 112 is an object or not based on the calculated likelihood. When the detection unit 101 detects an object, the detection result having information of the partial region in which the object is detected and a class described later is stored in the detection result DB 103 in association with the identification information of the image 111. As information of the partial region of the object, position information on the image 111 of the search window 112 determined to be an object is used. Here, as the position information, for example, when the search window 112 is a rectangle, the coordinate values ​​of the upper left and lower right vertices are used. In addition, there are three classes related to objects: class 1 representing a class of two-wheeled vehicles, class 2 representing a class of four-wheeled vehicles, and class 0 representing a class that is neither of the above. Furthermore, the identification information of the image 111 is, for example, the frame number of the image.

[0020] The detection unit 101 changes the position and size of the search window 112 and repeats the same operations as described above to thoroughly search for objects of different positions and sizes that exist in the image 111. In the example shown in FIG. 2, three objects are detected from the image 111. The detection unit 101 detects a first object using a search window with a position and size indicated by reference numeral 113, detects a second object using a search window with a position and size indicated by reference numeral 114, and detects a third object using a search window with a position and size indicated by reference numeral 115. In FIG. 2, the horizontal direction of the image is the X-axis and the vertical direction is the Y-axis. After completing the detection process for one image 111, the detection unit 101 repeats the same process for the next image.

[0021] FIG. 3 is an example of the detection result DB 103. The detection result DB 103 in this example records the detection results corresponding to the frame number. The detection results are composed of information on partial regions of objects and classes. For example, the detection result DB 103 records pairs of information on partial regions of three objects and classes corresponding to frame number 001. One of them is a pair of a partial region whose upper left vertex coordinates are (x1, y1) and whose lower right vertex coordinates are (x2, y2) and class 2, another is a pair of a partial region whose upper left vertex coordinates are (x3, y3) and whose lower right vertex coordinates are (x4, y4) and class 1, and the remaining one is a pair of a partial region whose upper left vertex coordinates are (x5, y5) and whose lower right vertex coordinates are (x6, y6) and class 1.

[0022] 3 has a column for recording a correction class for each detection result. The correction class corresponds to the class input by the reception unit 104. When the detection unit 101 outputs the detection result to the detection result DB 103, all of the columns for the correction class are NULL (indicated by - in FIG. 3).

[0023] The reception unit 104 visualizes the detection results of the images stored in the detection result DB 103, displays them on the display device 120, and receives input of corrections from the operator.

[0024] 4 shows an example of a reception screen 121 that the reception unit 104 displays on the display device 120. In this example, the reception screen 121 displays a graph 122 at the bottom of the screen, a slider 123 at the top thereof, and an image window 124 at the top thereof.

[0025] Graph 122 is a graph showing the relationship between a plurality of images including image 111 and the number of partial regions in which an object is detected in each screen. When the horizontal direction of reception screen 121 is the X axis and the vertical direction is the Y axis, the X axis of graph 122 shows the frame number of the image, and the Y axis shows the number of partial regions in which an object is detected, that is, the number of object detections. That is, reception unit 104 sets the X axis of graph 122 to a column in which input images corresponding to the detection results stored in detection result DB 103 are arranged in ascending or descending order of frame numbers. However, the X axis may be other than the column in which input images are arranged in ascending or descending order of frame numbers. For example, the X axis may be a column in which input images are arranged by the number of partial regions of objects (that is, the number of object detections). In this case, reception unit 104 sorts the input images of detection results 103 stored in detection result DB 103 in ascending or descending order by the number of object detections, and sets the column of sorted input images to the X axis of graph 122. Furthermore, although graph 122 is a bar graph in FIG. 4, the type of graph 122 is not limited to a bar graph, and may be another type, such as a line graph.

[0026] The slider 123 is a GUI (Graphical User Interface) for selecting a position on the X-axis of the graph 122, i.e., an image. The slider 123 includes a slide bar 123a, and the position on the X-axis of the graph 122 is selected by operating the slide bar 123a left and right with the input device 130.

[0027] The image window 124 displays, frame by frame, one image 111 selected from the multiple images by operating the slider 123. A display for emphasizing a partial region of a detected object is added to the image 111 displayed in the image window 124. In the example of Fig. 4, a rectangular frame 125 indicating the periphery of the partial region of the object is displayed. However, the display for emphasizing the partial region of the object is not limited to the rectangular frame 125, and any display form may be used, such as a display form in which the brightness of the entire partial region is made higher or lower than other regions, or a display form in which the entire partial region is shaded.

[0028] Furthermore, the display highlighting the partial region of the detected object is displayed in different display forms depending on the class of the detected object. In Fig. 4, the partial region of an object of class 1 is displayed with a dashed frame 125, and the partial region of an object of class 2 is displayed with a solid frame 125. In addition to varying the line type of frame 125, the display color of frame 125 may also be varied. Also, a numerical value indicating the class may be displayed near the partial image.

[0029] Generally, a GUI slider displays a scale and a label as a guide for the operator's operation. Instead of the scale and the label, a reception screen 121 shown in Fig. 4 displays a graph 122 indicating the number of object detections for each image. By displaying such a graph 122, the operator can easily select one image from multiple images based on the number of object detections.

[0030] The reception unit 104 also receives the selection of a partial region and the input of a class for the selected partial region by the operation of the input device 130 on the image 111 displayed on the display device 120. The reception unit 104 receives the selection of a partial region according to the position of the image 111 where the operation of the input device 130 is performed, and receives the input of a class according to the type of the operation. In this embodiment, the input device 130 is a mouse, and the types of operations are left click and right click. A left click means a single click of the left button of the mouse. A right click means a single click of the right button of the mouse. The reception unit 104 receives the selection of a partial region having within its region the position of the image 111 where the left click or right click is performed. The reception unit 104 also receives the input of class 0 for the selected partial region when the left click is performed, and receives the input of a class obtained by incrementing the current class when the right click is performed. Here, the class obtained by incrementing the current class means class 2 when the current class is class 1, class 0 when the current class is class 2, and class 1 when the current class is class 0. The receiving unit 104 records the received class for the selected partial region in a column for corrected class in the detection result DB 103 .

[0031] The generating unit 105 generates training data from the image of the partial region selected by the receiving unit 104 and the input class, and stores the training data in the training data memory 106. The generating unit 105 generates training data for each partial region in which any of class 0, class 1, and class 2 is recorded in the column for corrected class in the detection result DB 103 shown in Fig. 3. The training data is an input / output pair consisting of the image of the partial region or a feature amount extracted from the image of the partial region and the class recorded in the column for corrected class.

[0032] The learning unit 107 learns the dictionary 102 using the teacher data stored in the teacher data memory 106. The method of learning the dictionary 102 using the teacher data is widely known, so a description thereof will be omitted. The dictionary 102 is upgraded by learning using the teacher data. The detection unit 101 uses this upgraded dictionary 102 to detect objects again from each image input from the surveillance camera 110.

[0033] 5 is a flowchart showing the operation of this embodiment. Hereinafter, the operation of this embodiment will be described with reference to FIG.

[0034] The detection unit 101 of the object detection device 100 inputs images captured by the surveillance camera 110 one by one in chronological order (step S101). Next, the detection unit 101 detects an object from the input image using the dictionary 102 by the method described with reference to FIG. 2, and stores the detection result in the detection result DB 103 (step S102). If there is a next image captured by the surveillance camera 110 (step S103), the detection unit 101 returns to step S100 and repeats the same process as described above. As a result, the detection results of a plurality of images as shown in FIG. 3 are accumulated in the detection result DB 103. When the processing of the detection unit 101 for the series of images captured by the surveillance camera 110 is completed, the processing by the reception unit 104 is started.

[0035] The reception unit 104 displays a reception screen 121 as shown in Fig. 4 on the display device 120, and receives an input operation of the input device (step S104). The details of step S104 will be described later.

[0036] When the reception unit 104 finishes receiving the data, the generation unit 105 generates training data from the image of the partial region selected by the reception unit 104 and the input class (step S105). Next, the learning unit 107 learns the dictionary 102 using the training data stored in the training data memory 106 (step S106).

[0037] 6 is a flowchart showing the details of the operation of the reception unit 104 performed in step S104. The operation of the reception unit 104 will be described in detail below with reference to FIG.

[0038] The receiving unit 104 calculates the number of detected objects for each image from the detection result DB 103 (step S111). With reference to Fig. 3, three partial regions are detected from the image with frame number 001, so the receiving unit 104 sets the number of detected objects for frame number 001 to 3. In the same manner, the receiving unit 104 calculates the number of detected objects for images with other frame numbers.

[0039] Next, the reception unit 104 displays an initial reception screen 121 on the display device 120 (step S112). As shown in FIG. 4, the initial reception screen 121 displays a graph 122, a slider 123, and an image window 124. The graph 122 displays the number of detected objects for each frame number calculated in step S111. The slider 123 is a slide bar 123a placed at a predetermined position. The image window 124 displays an image of the object number on the X-axis of the graph 122 pointed to by the slide bar 123a. In addition, a partial area of ​​an object detected on the image displayed in the image window 124 is highlighted by a frame 125 of a line type according to the class of the partial area.

[0040] Next, the accepting unit 104 judges whether the slide bar 123a of the slider 123 has been operated (step S113), whether an image on the image window has been clicked (step S114), and whether an acceptance end condition has been established (step S115). The acceptance end condition may be, for example, that an acceptance end command has been input by the operator, or that no input operation has been performed for a certain period of time or more.

[0041] When the reception unit 104 detects a left click or right click on the image in the image window 124 (YES in step S114), it determines whether or not a partial area including the clicked position exists in the image (step S116). This determination is made by checking whether the coordinate values ​​of the clicked position are inside any partial area of ​​the image displayed in the image window 124. If there is no partial area including the clicked position (NO in step S116), the reception unit 104 ignores the click. On the other hand, if there is a partial area including the clicked position (YES in step S116), the reception unit 104 determines that the partial area including the clicked position has been selected, and determines whether the type of click is a left click or a right click to determine the input of a class (step S117).

[0042] If the click is a left click, the accepting unit 104 accepts input of class 0 for the selected partial region (step S118). Next, the accepting unit 104 updates the correction class corresponding to the selected partial region of the detection result in the detection result DB 103 to the accepted class 0 (step S119). Next, the accepting unit 104 makes the frame 125 displayed in the selected partial region of the image displayed in the image window 124 invisible (step S120).

[0043] Furthermore, in the case of right-clicking, the reception unit 104 receives an input of a class obtained by incrementing the current class of the selected partial region (step S121). The current class of the selected partial region is the class of the detection result when the corrected class corresponding to the selected partial region of the detection result in the detection result DB 103 is NULL, and is the class described in the corrected class when the corrected class is not NULL. Next, the reception unit 104 updates the corrected class corresponding to the selected partial region of the detection result in the detection result DB 103 to the incremented class (step S122). Next, the reception unit 104 updates the display of the frame 125 displayed in the selected partial region of the image displayed in the image window 124 (step S123). Specifically, if the incremented class is class 2, the reception unit 104 displays the frame 125 with a solid line. If the incremented class is class 0, the reception unit 104 hides the frame 125. Furthermore, if the class after the increment is class 1, the receiving unit 104 displays the frame 125 in a dashed line.

[0044] On the other hand, when the reception unit 104 detects that the slide bar 123a has been operated (step S113), it updates the image of the slide bar 123a on the reception screen 121 and the image window 124 (step S124). Specifically, the reception unit 104 moves the display position of the slide bar 123a in response to the input operation. The reception unit 104 also displays in the image window 124 an image of the object number on the X-axis of the graph 122 pointed to by the moved slide bar 123a.

[0045] Furthermore, the receiving unit 104 judges whether the partial area on the image displayed in the image window 124 has the same position and the same pre-input class as the partial area of ​​the image in which the class was previously input. If it is judged that they have the same position and the same pre-input class, the receiving unit 104 hides the frame 125 that highlights the partial area. For example, the positions of the third partial images in the frame numbers 001, 002, and 003 in FIG. 3 are all rectangles in the same position with the upper left vertex and the lower right vertex at (x5, y5) and (x6, y6), respectively, and the pre-input class is also the same class 1. Therefore, for example, after the class of the third partial image in the image of frame number 001 is corrected from class 1 to class 0, when the images of frame numbers 002 and 003 are displayed in the image window 124, the frame 125 is not displayed on the third partial image in the images of frame numbers 002 and 003. However, class 1 is still recorded as the class of those detection results.

[0046] In this way, according to the present embodiment, images for creating teacher data can be efficiently and easily selected based on the number of detected objects. More specifically, one image can be easily selected from a plurality of images based on the number of detected objects. This is because the reception unit 104 selects an appropriate input image for displaying the detection result on the display device 120. The reception unit 104 displays a graph 122 showing the relationship between the input image and the number of partial areas in which objects are detected on the display device 120, and selects an input image according to the position on the graph 122 received by operating the slider 123. In this way, the present embodiment can select an image to be processed based on the number of detected objects. Therefore, images in which many objects are detected can be processed preferentially according to the level of skill and preference of the operator. On the other hand, the present embodiment can easily perform preferential processing of images in which a small number of objects are detected, or preferential processing of images in which an intermediate number of objects are detected. In general, the total work time is shortened by preferential processing of images in which a large number of objects are detected. However, when there is a large amount of overlap between partial areas in an image, it is difficult for an operator who is not familiar with the process to confirm the overlap.

[0047] Furthermore, according to this embodiment, high-quality teacher data can be efficiently created because the receiving unit 104 displays the input image on the display device 120 with a display that highlights a partial region of the detected object, and receives the selection of a partial region and the input of a class for the selected partial region by a click, which is one operation of the input device 130.

[0048] In general, when a surveillance camera 110 with a fixed field of view captures images of two-wheeled or four-wheeled vehicles traveling on a road at a constant interval, the stationary object commonly appears at the same position in a number of consecutive frame images. For this reason, for example, when a puddle on a road is mistakenly detected as a two-wheeled vehicle, the same erroneous recognition result appears at the same position in a number of consecutive frame images. It is sufficient to correct one of the same erroneous recognition results to generate teacher data, and it is rather annoying if the same erroneous recognition result that has already been corrected is highlighted across a number of frame images. In this embodiment, the reception unit 104 hides the frame 125 that highlights the same position as a partial area of ​​an image in which a class was previously input and the partial area having the same pre-input class. This can eliminate the annoyingness.

[0049] [Modification of the first embodiment] Next, various modifications of the configuration of the first embodiment will be described.

[0050] In the first embodiment, the left click of the mouse gesture accepts input of class 0, and the right click accepts input of an increment of the class. However, various combinations of gesture types that accept input of class 0 and gesture types that accept input of an increment of the class are possible, as shown in the list of FIG.

[0051] 7, the accepting unit 104 accepts an input of class 0 by a left click of the mouse gesture, and an input of an increment of the class by a double click. A double click means an operation of single clicking the left mouse button twice in succession.

[0052] 7, the reception unit 104 receives an input of class 0 by a left click of the mouse gesture, and receives an input of an increment of the class by drag-and-drop. Drag-and-drop means an operation of moving (dragging) the mouse while pressing the left button of the mouse, and releasing (dropping) the left button at another location. When drag-and-drop is used, it is determined in step S114 of FIG. 6 whether drag-and-drop has been performed on the image window, and in step S116 it is determined, for example, whether there is a partial area that includes the start point of the drag and the drop point.

[0053] The numbers 1-1 and 1-2 use mouse gestures as in the first embodiment. On the other hand, the numbers 1-3 to 1-6 shown in FIG. 7 use touch gestures. A touch gesture means a touch operation performed on a touch panel using a part of the human body such as a fingertip. There are many types of touch gestures, such as tap (light tapping with a finger, etc.), double tap (tapping twice with a finger, etc.), flick (flicking or sweeping with a finger, etc.), swipe (tracing with a finger, etc.), pinch-in (pinching or narrowing with multiple fingers, etc.), and pinch-out (widening with multiple fingers, etc.). When using touch gestures, a touch panel is used as the input device 130. The use of a touch panel is also useful when the present invention is implemented in a mobile device such as a smartphone or tablet terminal.

[0054] In number 1-3 in FIG. 7, the reception unit 104 receives an input of class 0 by a flick, and receives an input of an increment of the class by a swipe. In number 1-4 in FIG. 7, the reception unit 104 receives an input of class 0 by a flick to the left, and receives an input of an increment of the class by a flick to the right. In number 1-5 in FIG. 7, the reception unit 104 receives an input of class 0 by a pinch-in, and receives an input of an increment of the class by a pinch-out. In number 1-6 in FIG. 7, the reception unit 104 receives an input of class 0 by a tap, and receives an input of an increment of the class by a double tap. These are examples, and any other combination of gestures may be used.

[0055] In the above first embodiment and numbers 1-1 to 1-6 in Fig. 7, the reception unit 104 receives an input of class 0 by a first type of gesture, and receives an input of an increment of the class by a second type of gesture. However, the reception unit 104 may receive an input of a decrement instead of an increment. That is, in the above first embodiment and numbers 1-1 to 1-6 in Fig. 7, the reception unit 104 may receive an input of class 0 by a first type of gesture, and receives an input of a decrement of the class by a second type of gesture.

[0056] In the first embodiment, the receiving unit 104 receives input of class 0 by left-clicking the mouse gesture, and receives input of incrementing the class by right-clicking. However, classes and operations may be associated one-to-one, and the receiving unit 104 may receive input of class 0, class 1, and class 2 by one specific operation each. As shown in the list of FIG. 8, various combinations of gesture types for receiving input of each class are possible.

[0057] For example, in the case of number 1-11 shown in FIG. 8, the reception unit 104 receives an input of class 0 by a left click of the mouse gesture, receives an input of class 1 by a right click, and receives an input of class 2 by a double click. In addition, in the case of number 1-12 shown in FIG. 8, the reception unit 104 receives inputs of class 0 and class 1 by a mouse gesture similar to that of number 1-11, and receives an input of class 2 by drag and drop. The above is an example using mouse gestures. In contrast, in the cases of numbers 1-13 to 1-15 shown in FIG. 8, touch gestures are used. For example, in the case of number 1-13 shown in FIG. 8, the reception unit 104 receives an input of class 0 by a flick, receives an input of class 1 by a tap, and receives an input of class 2 by a double tap. In addition, in the case of number 1-14 shown in FIG. 8, the reception unit 104 receives an input of class 0 by a left flick, receives an input of class 1 by an upward flick, and receives an input of class 2 by a downward flick. In addition, in the case of number 1-15 shown in FIG. 8, the accepting unit 104 accepts an input of class 0 by tapping, an input of class 1 by pinching in, and an input of class 2 by pinching out.

[0058] In the above first embodiment, the number of classes was three (class 0, class 1, class 2), but the present invention can also be applied to a two-class object detection device with two classes (class 0, class 1), and a multi-class object detection device with four or more classes.

[0059] [Second embodiment] In the second embodiment of the present invention, an object detection device is described that allows selection of a partial region of an object that cannot be mechanically detected from an input image and input of a class for that partial region by operating an input device on the input image. This embodiment is different from the first embodiment in the function of the reception unit 104, but is otherwise basically the same as the first embodiment. Below, the function of the reception unit 104 suitable for the second embodiment will be described in detail with reference to FIG. 1, which is a block diagram of the first embodiment.

[0060] The reception unit 104 visualizes the image detection result DB 103, displays it on the display device 120, and receives input of corrections from the operator.

[0061] 9 shows an example of a reception screen 126 that the reception unit 104 displays on the display device 120. The reception screen 126 in this example differs from the reception screen 121 shown in FIG. 4 only in that a rectangular frame indicating the periphery of a partial region is not displayed at the location of the two-wheeled vehicle in the image 111 displayed in the image window 124. The rectangular frame is not displayed because the detection unit 101 failed to detect the two-wheeled vehicle at that location when it detected an object from the image 111 using the dictionary 102. In order to improve such missed detections, the object detection device needs to create teacher data that is an input / output pair between the partial region of the two-wheeled vehicle in the image 111 and its class, and learn the dictionary 102.

[0062] The reception unit 104 receives the selection of a partial region and the input of a class for the selected partial region through an operation of the input device 130 on the image 111 displayed on the display device 120. In this embodiment, the selection of a partial region means both the selection of an existing partial region and the creation of a new partial region. The reception unit 104 receives the selection of a partial region based on the position on the image 111 where the operation of the input device 130 was performed, and receives the input of a class based on the type of operation. In this embodiment, the input device 130 is a mouse, and the types of operations are left click, right click, drag and drop in the lower right direction, and drag and drop in the lower left direction.

[0063] Drag-and-drop in the lower-right direction means, as shown by the arrow in Fig. 10A, moving (dragging) the mouse from a certain location (start point) a on image 111 while pressing the left button, and then releasing (dropping) the left button at another location (end point) b located to the lower-right of start point a. Drag-and-drop in the lower-left direction means, as shown by the arrow in Fig. 10B, moving (dragging) the mouse from a certain location (start point) a on image 111 while pressing the left button, and then releasing (dropping) the left button at another location (end point) b located to the lower-left of start point a.

[0064] The reception unit 104 receives a selection of a partial region having within it the position of the image 111 that has been left-clicked or right-clicked. Furthermore, the reception unit 104 receives an input of class 0 for the selected partial region when the user left-clicks, and receives an input of a class obtained by incrementing the current class when the user right-clicks. Here, the class obtained by incrementing the current class means class 2 when the current class is class 1, class 0 when the current class is class 2, and class 1 when the current class is class 0. The reception unit 104 records the received class for the selected partial region in the column of corrected class of the detection result 103. The above is the same as when the user left-clicks or right-clicks in the first embodiment.

[0065] 10A, and accepts input of class 1 for the partial region. Furthermore, when the reception unit 104 detects a drag-and-drop in the lower-left direction, it calculates a rectangular region with the start point a as the upper-left vertex and the end point b as the lower-right vertex as shown by the dashed line in Fig. 10A, and accepts input of class 2 for the partial region. Furthermore, when the reception unit 104 detects a drag-and-drop in the lower-left direction, it calculates a rectangular region with the start point a as the upper-right vertex and the end point b as the lower-left vertex as shown by the dashed line in Fig. 10B, and accepts input of class 2 for the partial region. Furthermore, the reception unit 104 records information on the newly calculated partial region and its class in the detection result DB 103.

[0066] Fig. 11 is a flowchart showing the details of the operation of the reception unit 104. In Fig. 11, steps S211 to S224 are the same as steps S111 to S124 in Fig. 6. Hereinafter, the operation of the reception unit 104 will be described in detail with reference to Fig. 11, focusing on the differences from Fig. 6.

[0067] When the reception unit 104 detects a drag-and-drop on the image of the image window 124 (YES in step S225), it determines the direction of the drag-and-drop (step S226). If the drag-and-drop is in the lower right direction, the reception unit 104 calculates a rectangular partial area with the upper left vertex as the starting point and the lower right vertex as the ending point, as described with reference to FIG. 10A, and receives the input of class 1 (step S227). Next, the reception unit 104 records the pair of the calculated partial area and class 1 as one detection result in the detection result DB 103 (step S228). Next, the reception unit 104 updates the image window 124 of the reception screen 126 (step S229). Then, the reception unit 104 returns to the process of step S213.

[0068] If the drag-and-drop is in the lower left direction (step S226), the accepting unit 104 calculates a rectangular partial area with the start point being the upper right vertex and the end point being the lower left vertex as described with reference to FIG. 10B, and accepts the input of class 2 (step S230). Next, the accepting unit 104 records the pair of the calculated partial area and class 2 as one detection result in the detection result DB 103 (step S231). Next, the accepting unit 104 updates the image window 124 of the accepting screen 126 (step S232). Then, the accepting unit 104 returns to the process of step S213.

[0069] 12A and 12B are explanatory diagrams of step S228 by the receiving unit 104. FIG. 12A shows the detection result DB 103 generated by the detection unit 101, and FIG. 12B shows the detection result DB 103 after the receiving unit 104 adds a record of a pair of a newly calculated partial region and class 1. The newly added partial region is a rectangle with an upper left vertex (x3, y3) and a lower right vertex (x4, y4), and its class is class 1. In this way, the class of the added partial region is recorded in the column of the corrected class. In this way, the generating unit 105 generates teacher data from the added partial region and class. In the case of step S231 by the receiving unit 104, it is different in that class 2 is recorded in the corrected class.

[0070] 13A and 13B are explanatory diagrams of step S229 by the reception unit 104. Fig. 13A shows the image window 124 before updating, and Fig. 13B shows the image window 124 after updating. A rectangular frame 125 drawn with dashed lines to highlight the newly added partial region is drawn in the updated image window 124. In the case of step S232 by the reception unit 104, a rectangular frame drawn with solid lines to highlight the newly added partial region is drawn in the updated image window 124, which is different.

[0071] As described above, according to the present embodiment, images for creating teacher data can be efficiently and simply selected based on the number of detected objects, and high-quality teacher data can be efficiently created. This is because the reception unit 104 displays the input image on the display device 120 with a display that highlights the partial region of the detected object, and receives the selection of the partial region and the input of the class for the selected partial region by a click, which is one operation of the input device 130. Also, the reception unit 104 detects one operation of dragging and dropping in the lower right or lower left direction on the image, and receives the calculation of a new partial region and the input of class 1 or class 2.

[0072] [Modification of the second embodiment] Next, various modified examples in which the configuration of the second embodiment of the present invention is changed will be described.

[0073] In the second embodiment, the receiving unit 104 receives input of class 0 by left-clicking the mouse gesture, receives input of an increment of the class by right-clicking, calculates a partial area of ​​class 1 by dragging and dropping in the lower right direction, and calculates a partial area of ​​class 2 by dragging and dropping in the lower left direction. However, various combinations of gesture types are possible, as exemplified in the list in Fig. 14. The gesture types include receiving input of class 0, receiving input of an increment of the class, receiving calculation of a partial area of ​​class 1, and receiving calculation of a partial area of ​​class 2.

[0074] 14, the receiving unit 104 receives an input of class 0 by a left click of the mouse gesture, and receives an input of an increment of the class by a double click. Furthermore, the receiving unit 104 receives a calculation of a partial area of ​​class 1 by dragging and dropping in the lower right direction, and receives a calculation of a partial area of ​​class 2 by dragging and dropping in the lower left direction.

[0075] In addition, in the case of number 2-2 shown in FIG. 14, the receiving unit 104 receives an input of class 0 by left-clicking the mouse gesture, and receives an input of increment of the class by right-clicking. Furthermore, the receiving unit 104 receives a calculation of a partial area of ​​class 1 by dragging and dropping in the upper left direction, and receives a calculation of a partial area of ​​class 2 by dragging and dropping in the upper right direction. Here, the dragging and dropping in the upper left direction is a dragging and dropping in the opposite direction to the arrow in FIG. 10A, and means an operation of moving (dragging) the mouse from a certain place (start point) b on the image 111 while pressing the left button of the mouse, and releasing (dropping) the left button at another place (end point) a located in the upper left direction of the start point b. Furthermore, the dragging and dropping in the upper right direction is a dragging and dropping in the opposite direction to the arrow in FIG. 10B, and means an operation of moving (dragging) the mouse from a certain place (start point) b on the image 111 while pressing the left button of the mouse, and releasing (dropping) the left button at another place (end point) a located in the upper right direction of the start point b.

[0076] 14, the operation for accepting the calculation of the partial area of ​​class 1 and class 2 is different from that for number 2-2. In number 2-3, the accepting unit 104 accepts the calculation of the partial area of ​​class 1 by a left double click, and accepts the calculation of the partial area of ​​class 2 by a right double click. Here, a left double click means clicking the left button of the mouse twice in succession. Also, a right double click means clicking the right button of the mouse twice in succession.

[0077] FIG. 15 is an explanatory diagram of a method in which the reception unit 104 calculates a partial area of ​​class 2 from the position of a right double click performed on the image 111. When the reception unit 104 detects that a right double click has been performed at a point c on the image 111, the reception unit 104 assumes that the center of gravity of a typical class 2 object (a four-wheeled vehicle in this embodiment) 131 is present at a position that coincides with point c, and calculates a circumscribing rectangle 132 of the object as a partial area. When an object is captured by a surveillance camera 110 with a fixed imaging field of view, there is a correlation between each point of the object and each point on the image, so that the circumscribing rectangle of an object of a typical size having a center of gravity at a certain point on the image is almost uniquely determined. Furthermore, the circumscribing rectangle of a typical class 1 object (a two-wheeled vehicle in this embodiment) having a center of gravity at a certain point on the image is also almost uniquely determined. In this way, the reception unit 104 calculates a partial area of ​​class 1 from the position of a left double click performed on the image 111 by a similar method.

[0078] In the example of FIG. 15, the operator is configured to right-double-click or left-double-click the center of gravity of an object via the input device 130, but the center of gravity is not limited to the center of gravity, and any predetermined position may be used. For example, the center of the front wheel of a two-wheeled vehicle, or the middle between the front and rear wheels of a four-wheeled vehicle may be right-double-clicked or left-double-clicked. In addition, in the case of an object detection device that detects people, the operator is configured to left-double-click the person's head via the input device 130. In this case, the receiving unit 104 estimates a person area from the head position, and calculates a circumscribing rectangle of the estimated person area as a partial area.

[0079] The above numbers 2-1 to 2-3 use mouse gestures as in the second embodiment, whereas numbers 2-4 to 2-7 shown in Fig. 14 use touch gestures as follows.

[0080] In number 2-4 of Figure 14, the receiving unit 104 accepts input of class 0 by a flick, accepts input of a class increment by a tap, accepts calculation of a partial area of ​​class 1 by a swipe in the lower right direction, and accepts calculation of a partial area of ​​class 2 by a swipe in the lower left direction.

[0081] A swipe in the lower right direction means an operation of tracing with a fingertip from a certain place (starting point) a on the image 111 to another place (ending point) b located in the lower right direction of the starting point a, as shown by an arrow in FIG. 16A. A swipe in the lower left direction means an operation of tracing with a fingertip from a certain place (starting point) a on the image 111 to another place (ending point) b located in the lower left direction of the starting point a, as shown by an arrow in FIG. 16B. When detecting a swipe in the lower right direction, the reception unit 104 calculates a rectangular area with the starting point a as the upper left vertex and the ending point b as the lower right vertex as a partial area, as shown by a dashed line in FIG. 16A, and accepts an input of class 1 for the partial area. When detecting a swipe in the lower left direction, the reception unit 104 calculates a rectangular area with the starting point a as the upper right vertex and the ending point b as the lower left vertex as a partial area, as shown by a dashed line in FIG. 16B, and accepts an input of class 2 for the partial area.

[0082] In number 2-5 of Figure 14, the reception unit 104 accepts input of class 0 with a left flick, accepts input of a class increment with a right flick, accepts calculation of a partial area of ​​class 1 with a downward swipe, and accepts calculation of a partial area of ​​class 2 with an upward swipe.

[0083] A downward swipe means an operation of tracing with a fingertip from a certain place (start point) a on the image 111 to another place (end point) b located below the start point a, as shown by the arrow in FIG. 17A. An upward swipe means an operation of tracing with a fingertip from a certain place (start point) b on the image 111 to another place (end point) a located above the start point b, as shown by the arrow in FIG. 17B. When the reception unit 104 detects a downward swipe, it calculates a rectangle as shown by the dashed line in FIG. 17A as a partial area of ​​class 1. The rectangle is line-symmetrical with respect to the swipe line, and the vertical length is equal to the swipe length. The horizontal length is calculated by multiplying the swipe length by a predetermined ratio. As the predetermined ratio, for example, the ratio of the horizontal length to the vertical length of a circumscribing rectangle of a class 1 object of a typical shape (a two-wheeled vehicle in this embodiment) can be used. Furthermore, when the reception unit 104 detects an upward swipe, it calculates a rectangle as shown by the dashed line in FIG. 17B as a partial area of ​​class 2. The rectangle is symmetrical about the swipe line, and the vertical length is equal to the length of the swipe. The horizontal length is calculated by multiplying the length of the swipe by a predetermined ratio. As the predetermined ratio, the ratio of the horizontal length to the vertical length of the circumscribing rectangle of a class 2 object of a typical shape (a four-wheeled automobile in this embodiment) can be used.

[0084] In number 2-6 of Figure 14, the receiving unit 104 accepts input of class 0 by pinching in, accepts input of a class increment by pinching out, accepts calculation of a partial area of ​​class 1 by swiping in the lower right direction, and accepts calculation of a partial area of ​​class 2 by swiping in the lower left direction.

[0085] 14, the accepting unit 104 accepts an input of class 0 by tapping, an input of an increment of the class by double tapping, a calculation of a partial area of ​​class 1 by swiping in the lower right direction, and a calculation of a partial area of ​​class 2 by swiping in the lower left direction. These are merely examples, and any other combination of gestures may be used.

[0086] In the above second embodiment and in the numbers 2-1 to 2-7 in FIG. 14, the reception unit 104 receives an input of class 0 by a gesture of the first type, and receives an input of an increment of the class by a gesture of the second type. However, the reception unit 104 may receive an input of a decrement of the class instead of an increment. That is, in the above second embodiment and in the numbers 2-1 to 2-7 in FIG. 14, the reception unit 104 may receive an input of a decrement of the class by a gesture of the second type.

[0087] In the above second embodiment, the receiving unit 104 receives input of class 0 by left-clicking the mouse gesture, and receives input of incrementing the class by right-clicking. However, classes and operations may be associated one-to-one, and the receiving unit 104 may receive input of class 0, class 1, and class 2 by one specific operation. The gesture types for receiving input of each class and the gesture types for receiving calculation of partial areas of each class may be combined in various ways, as exemplified in the list of FIG. 18.

[0088] 18, the receiving unit 104 receives input of class 0 by a left click of the mouse gesture, and receives input of class 1 by a right click. Furthermore, the receiving unit 104 receives input of class 2 by a double click, receives calculation of a partial area of ​​class 1 by dragging and dropping in the lower right direction, and receives calculation of a partial area of ​​class 2 by dragging and dropping in the lower left direction.

[0089] 18 differs from number 2-11 in the gesture types for accepting calculation of partial areas of class 1 and class 2. In number 2-12, the accepting unit 104 accepts calculation of a partial area of ​​class 1 by drag-and-drop in the upper left direction, and accepts calculation of a partial area of ​​class 2 by drag-and-drop in the upper right direction.

[0090] Also, in number 2-13 shown in FIG. 18, the reception unit 104 accepts input of class 0 by a touch gesture flick, accepts input of class 1 by a tap, accepts input of class 2 by a double tap, accepts calculation of a partial area of ​​class 1 by a swipe in the lower right direction, and accepts calculation of a partial area of ​​class 2 by a swipe in the lower left direction.

[0091] 18 differs from number 2-13 in the gesture types that accept inputs of classes 0, 1, and 2. In number 2-14, the accepting unit 104 accepts an input of class 0 by a leftward flick, an input of class 1 by an upward flick, and an input of class 2 by a downward flick.

[0092] Also, in number 2-15 shown in FIG. 18, the receiving unit 104 accepts input of class 0 by tapping, accepts input of class 1 by pinching in, accepts input of class 2 by pinching out, accepts calculation of a partial area of ​​class 1 by swiping downward, and accepts calculation of a partial area of ​​class 2 by swiping upward.

[0093] In the above second embodiment, the number of classes was three (class 0, class 1, class 2), but the present invention can also be applied to a two-class object detection device with two classes (class 0, class 1), and a multi-class object detection device with four or more classes.

[0094] [Third embodiment] In the third embodiment of the present invention, an object detection device is described that accepts inputs of different classes even for the same operation depending on whether or not there is an existing partial area that has the position of the image where the operation was performed within the area. This embodiment differs from the second embodiment in the function of the reception unit 104, but is otherwise basically the same as the second embodiment. Below, the function of the reception unit 104 suitable for the third embodiment will be described in detail with reference to FIG. 1, which is a block diagram of the first embodiment.

[0095] The reception unit 104 visualizes the image detection result DB 103, displays it on the display device 120, and receives input of corrections from the operator.

[0096] The reception unit 104 receives the selection of a partial region and the input of a class for the selected partial region by the operation of the input device 130 on the image 111 displayed on the display device 120. Here, in this embodiment, the selection of a partial region means both the selection of an existing partial region and the selection of a new partial region. The reception unit 104 receives the selection of a partial region based on the position of the image 111 where the operation of the input device 130 was performed, and receives the input of a class based on the type of the operation. Also, in this embodiment, the reception unit 104 receives input of a different class even for the same operation depending on whether or not there is an existing partial region that has the position of the image where the operation was performed within the region. In this embodiment, the input device 130 is a mouse, and the types of one operation are a left click and a right click.

[0097] The reception unit 104 receives a selection of an existing partial region having within it the position of the image 111 that has been left-clicked or right-clicked. Furthermore, the reception unit 104 receives an input of class 0 for the selected partial region when the user left-clicks, and receives an input of a class obtained by incrementing the current class when the user right-clicks. Here, the class obtained by incrementing the current class means class 2 when the current class is class 1, class 0 when the current class is class 2, and class 1 when the current class is class 0. The reception unit 104 records the received class for the selected partial region in the column of modified class in the detection result DB 103. The above is the same as when the user left-clicks or right-clicks in the first embodiment.

[0098] Furthermore, if there is no existing partial area that has the position of the left-clicked or right-clicked image 111 within its area, the receiving unit 104 calculates a new rectangular partial area based on the clicked position, and receives an input of class 1 for the partial area if it is a left click, or class 2 for the partial area if it is a right click. The method of calculating a new partial area based on the clicked position can be the same as the method of calculating a partial area of ​​an object based on the double-clicked position described with reference to Fig. 15.

[0099] FIG. 19 is a flowchart showing the details of the operation of the reception unit 104. In FIG. 19, steps S211 to S224 and S226 to S232 are almost the same as steps S211 to S224 and S226 to S232 in FIG. 11. FIG. 19 differs from FIG. 11 in that if it is determined in step S216 that there is no existing partial area that includes the clicked position, the process proceeds to step S226. Note that step S226 in FIG. 11 determines the direction (type) of drag and drop, but step S226 in FIG. 19 determines the type of click. Hereinafter, the operation of the reception unit 104 will be described in detail with reference to FIG. 19, focusing on the differences from FIG. 11.

[0100] The receiving unit 104 detects a left click or a right click on the image in the image window 124 (YES in step S214). At this time, if the receiving unit 104 determines that a partial area including the click position does not exist in the image (NO in step S216), it determines whether the type of click is a left click or a right click in order to determine the input of a class (step S226).

[0101] If the click is a left click, the accepting unit 104 calculates a new rectangular partial area and accepts the input of class 1 (step S227). Next, the accepting unit 104 records the pair of the calculated partial area and class 1 as one detection result in the detection result DB 103 (step S228). Next, the accepting unit 104 updates the image window 124 of the acceptance screen 126 (step S229). Then, the accepting unit 104 returns to the process of step S213.

[0102] Furthermore, if the click is a right click (step S226), the accepting unit 104 calculates a new rectangular partial area and accepts input of class 2 (step S230). Next, the accepting unit 104 records the pair of the calculated partial area and class 2 as one detection result in the detection result DB 103 (step S231). Next, the accepting unit 104 updates the image window 124 of the accepting screen 126 (step S232). Then, the accepting unit 104 returns to the process of step S213.

[0103] As described above, according to the present embodiment, images for creating teacher data can be efficiently and simply selected based on the number of detected objects, and high-quality teacher data can be efficiently created. This is because the reception unit 104 displays the input image on the display device 120 with a display that highlights the partial region of the detected object, and receives the selection of a partial region and the input of a class for the selected partial region by a single operation of the input device 130, that is, a click. Also, the reception unit 104 detects a single operation of a left click or a right click at a position that does not overlap with an existing partial region, and receives the calculation of a new partial region and the input of class 1 or class 2.

[0104] [Modification of the third embodiment] In the above third embodiment, two types of gestures, left click and right click, are used, but other types of mouse gesture combinations and touch gesture combinations can be used.

[0105] [Fourth embodiment] In the fourth embodiment of the present invention, an object detection device is described that allows input of a correct class for an erroneously recognized partial region and selection of a correct partial region of an object by operating an input device on an input image. This embodiment is different from the second embodiment in the function of the reception unit 104, but is otherwise basically the same as the second embodiment. Below, the function of the reception unit 104 suitable for the fourth embodiment will be described in detail with reference to FIG. 1, which is a block diagram of the first embodiment.

[0106] The reception unit 104 visualizes the image detection result DB 103, displays it on the display device 120, and receives input of corrections from the operator.

[0107] FIG. 20 shows an example of a reception screen 127 that the reception unit 104 displays on the display device 120. The reception screen 127 in this example differs from the reception screen 121 shown in FIG. 4 only in the position of a rectangular frame indicating the periphery of the partial region of the four-wheeled vehicle in the image 111 displayed in the image window 124. The reason for the difference in the position of the rectangular frame is that when the detection unit 101 detects an object from the image 111 using the dictionary 102, it erroneously detects a puddle on the road as a part of the partial region of the object. In this example, the partial region is erroneously detected, but the class is correctly determined. However, there are also cases where both the partial region and the class are erroneously detected. In order to improve such erroneous detection, it is necessary to create teacher data that is an input / output pair between the erroneously detected partial region in the image 111 and its class 0, and teacher data that is an input / output pair between the correct partial region of the four-wheeled vehicle in the image 111 and its class 2, and to learn the dictionary 102.

[0108] The reception unit 104 receives the selection of a partial region and the input of a class for the selected partial region by the operation of the input device 130 on the image 111 displayed on the display device 120. Here, in this embodiment, the selection of a partial region means both the selection of an existing partial region and the selection of a new partial region. The reception unit 104 receives the selection of a partial region based on the position of the image 111 where the operation of the input device 130 is performed, and receives the input of a class based on the type of the operation. Also, in this embodiment, when a selection operation of a new partial region that partially overlaps with an existing partial region is performed, the reception unit 104 receives the input of class 0 for the existing partial region and receives the input of a class according to the type of operation for the new partial region. In this embodiment, the input device 130 is a mouse, and the types of one operation are a left click, a right click, a drag and drop in the lower right direction, and a drag and drop in the lower left direction.

[0109] The reception unit 104 receives a selection of a partial region having within it the position of the image 111 that has been left-clicked or right-clicked. Furthermore, the reception unit 104 receives an input of class 0 for the selected partial region when the user left-clicks, and receives an input of a class obtained by incrementing the current class when the user right-clicks. Here, the class obtained by incrementing the current class means class 2 when the current class is class 1, class 0 when the current class is class 2, and class 1 when the current class is class 0. The reception unit 104 records the received class for the selected partial region in the column of corrected class in the detection result DB 103. The above is the same as when the user left-clicks or right-clicks in the first embodiment.

[0110] Furthermore, when the reception unit 104 detects a drag-and-drop in the lower right direction, it calculates a rectangular region with the start point a as the upper left vertex and the end point b as the lower right vertex as a partial region, as shown by the dashed line in FIG. 10A, and accepts an input of class 1 for the partial region. Furthermore, when the reception unit 104 detects a drag-and-drop in the lower left direction, it calculates a rectangular region with the start point a as the upper right vertex and the end point b as the lower left vertex as a partial region, as shown by the dashed line in FIG. 10B, and accepts an input of class 2 for the partial region. Furthermore, the reception unit 104 records the newly calculated partial region and information on its class in the detection result DB 103 as one detection result. The above is the same as when drag-and-drop is performed in the lower right direction and the lower left direction in the second embodiment.

[0111] Furthermore, the receiving unit 104 receives an input of class 0 for an existing partial region that partially overlaps with a newly calculated partial region. For example, as shown in Fig. 21, when a drag-and-drop is performed in the lower left direction (or lower right direction) on an image 111 in which a frame 125 highlighting a partial region of class 2 is displayed, the receiving unit 104 calculates a partial region 125a that partially overlaps with the frame 125, and receives an input of class 0 for the existing partial region related to the frame 125.

[0112] Fig. 22 is a flowchart showing the details of the operation of the reception unit 104. In Fig. 22, steps S311 to S327 and S330 are the same as steps S211 to S227 and S230 in Fig. 11. Hereinafter, the operation of the reception unit 104 will be described in detail with reference to Fig. 22, focusing on the differences from Fig. 11.

[0113] When the reception unit 104 receives an input of class 1 for the newly calculated partial region (step S327), it judges whether or not an existing partial region partially overlapping the new partial region exists on the image 111 (step S341). This judgment is performed by investigating whether or not a detection result of an existing partial region partially overlapping the new partial region exists in the detection result DB 103 related to the image 111. If such an existing partial region exists, the reception unit 104 receives an input of class 0 for the partial region (step S342) and proceeds to step S343. If such an existing partial region does not exist, the reception unit 104 skips the process of step S342 and proceeds to step S343. In step S343, the reception unit 104 updates the detection result DB 103 according to the reception results of steps S327 and S342. That is, the reception unit 104 records a pair of the newly calculated partial region and class 1 as one detection result in the detection result DB 103 based on the reception result of step S327. Furthermore, the receiving unit 104 records class 0 as the correction class of the existing partial region based on the reception result of step S342. Next, the receiving unit 104 updates the image 111 on the image window 124 in accordance with the update of the detection result DB 103 in step S343 (step S344). Then, the receiving unit 104 returns to the process of step S313.

[0114] Furthermore, when the reception unit 104 receives an input of class 2 for the newly calculated partial region (step S330), it determines whether or not an existing partial region that partially overlaps the new partial region exists on the image 111 (step S345). If such an existing partial region exists, the reception unit 104 receives an input of class 0 for the partial region (step S346) and proceeds to step S347. If such an existing partial region does not exist, the reception unit 104 skips the process of step S346 and proceeds to step S347. In step S347, the reception unit 104 updates the detection result DB 103 according to the reception results of steps S330 and S346. That is, the reception unit 104 records a pair of the newly calculated partial region and class 2 as one detection result in the detection result DB 103 based on the reception result of step S330. Furthermore, the reception unit 104 records class 0 in the correction class of the existing partial region based on the reception result of step S346. Next, the reception unit 104 updates the image 111 on the image window 124 in accordance with the update of the detection result DB 103 in step S347 (step S348). Then, the reception unit 104 returns to the process of step S313.

[0115] 23A and 23B are explanatory diagrams of step S347 by the receiving unit 104. FIG. 23A shows the detection result DB 103 generated by the detection unit 101, and FIG. 23B shows the detection result DB 103 after it has been updated by the receiving unit 104. In FIG. 23B, a rectangular partial area with an upper left vertex (x7, y7) and a lower right vertex (x8, y8) is newly added as class 2. Also, the class of an existing rectangular partial area with an upper left vertex (x1, y1) and a lower right vertex (x2, y2) is corrected from class 2 to class 0. In the case of step S343 by the receiving unit 104, the difference is that the corrected class of the newly added partial area becomes class 1.

[0116] 24A and 24B are explanatory diagrams of step S348 by reception unit 104. Fig. 24A shows image window 124 before updating, and Fig. 24B shows image window 124 after updating. In updated image window 124, rectangular frame 125 drawn with solid lines to highlight the newly added partial region is drawn, and the frame that was displayed in image window 124 before updating and that partially overlaps with the rectangular frame is not displayed. In the case of step S344 by reception unit 104, the difference is that in updated image window 124, a rectangular frame that highlights the newly added partial region is drawn with dashed lines.

[0117] As described above, according to the present embodiment, images for creating teacher data can be efficiently and easily selected based on the number of detected objects, and high-quality teacher data can be efficiently created. This is because the reception unit 104 displays the input image on the display device 120 with a display that highlights the partial region of the detected object, and receives the selection of the partial region and the input of the class for the selected partial region by a click, which is one operation of the input device 130. In addition, the reception unit 104 detects one operation of dragging and dropping in the lower right or lower left direction on the image, and receives the calculation of a new partial region and the input of class 1 or class 2. In addition, the reception unit 104 receives the input of class 0 for an existing partial region that partially overlaps with the newly calculated partial region.

[0118] [Modification of the fourth embodiment] The configuration of the fourth embodiment of the present invention can be modified in the same manner as the example described above as the modified example of the second embodiment.

[0119] [Fifth embodiment] A fifth embodiment of the present invention will be described. With reference to Fig. 25, an object detection device 1 according to this embodiment detects an object from an input image 2. The object detection device 1 has a dictionary 10, and also has, as main functional units, a detection unit 11, a reception unit 12, a generation unit 13, and a learning unit 14. Furthermore, the object detection device 1 is connected to a display device 3 and an input device 4.

[0120] The detection unit 11 detects an object from each of the multiple input images 2 using the dictionary 10. The reception unit 12 displays a graph indicating the relationship between the input image 2 and the number of partial regions in which objects are detected on the display device 3, and displays one of the multiple input images on the display device 3 according to a position on the graph received by operation of the input device 4 for creating teacher data. The generation unit 13 generates teacher data from the input image. The learning unit 14 learns the dictionary 10 using the teacher data generated by the generation unit 13, and upgrades the dictionary 10.

[0121] Next, the operation of the object detection device 1 according to this embodiment will be described. The detection unit 11 of the object detection device 1 detects an object from the input image 2 using the dictionary 10, and notifies the reception unit 12 of the detection result. Based on the notified detection result, the reception unit 12 displays a graph showing the relationship between the input image 2 and the number of partial regions in which the object is detected on the display device 3. Furthermore, the reception unit 12 displays one of the multiple input images on the display device 3 according to the position on the graph received by the operation of the input device 4 to create teacher data. The generation unit 13 generates teacher data from the displayed input image, and notifies the learning unit 14. The learning unit 14 learns the dictionary 10 using the teacher data generated by the generation unit 13, and upgrades the dictionary 10.

[0122] As described above, according to the present embodiment, images for creating teacher data can be efficiently and easily selected based on the number of detected objects. This is because the receiving unit 12 displays a graph showing the relationship between the input image 2 and the number of partial regions in which objects are detected on the display device 3. Furthermore, the receiving unit 12 displays one of the multiple input images on the display device 3 for creating teacher data according to the position on the graph received by operating the input device 4.

[0123] This embodiment is based on the above-mentioned configuration, but various additions and modifications can be made as described below.

[0124] The reception unit 12 may be configured to display, on the display device 3, a slide bar for selecting a position on the graph.

[0125] The receiving unit 12 may be configured to treat a column in which a plurality of input images 2 are arranged in the order of frame numbers as one axis of a graph.

[0126] The receiving unit 12 may be configured to arrange a plurality of input images 2 in accordance with the number of partial regions of an object, and use the resulting column as one axis of the graph.

[0127] The receiving unit 12 may be configured to receive a selection of a partial region based on the position of the input image 2 where one operation has been performed.

[0128] [Sixth embodiment] An object detection device 300 according to the sixth embodiment of the present invention will be described with reference to Fig. 27. The object detection device 300 includes a detection unit 301, a reception unit 302, a generation unit 303, and a learning unit 304.

[0129] The detection unit 301 detects an object from each of a plurality of input images using a dictionary.

[0130] The receiving unit 302 displays on a display device a graph showing the relationship between the input image and the number of partial areas in which objects are detected, and displays on the display device one of the multiple input images according to the position on the graph received by operation of the input device for creating teacher data.

[0131] The generation unit 303 generates training data from the input image.

[0132] The learning unit 304 learns the dictionary using the training data.

[0133] According to the present embodiment, images for creating teacher data can be efficiently and easily selected based on the number of detected objects. This is because the receiving unit 302 displays a graph on the display device that indicates the relationship between the input image and the number of partial regions in which objects are detected. Furthermore, the receiving unit 302 displays one of the multiple input images on the display device for creating teacher data according to the position on the graph received by operating the input device.

[0134] The present invention has been described above by taking the above-mentioned embodiment as an exemplary example. However, the present invention is not limited to the above-mentioned embodiment. That is, the present invention can be applied in various aspects that can be understood by a person skilled in the art within the scope of the present invention.

[0135] This application claims priority based on Japanese Patent Application No. 2015-055927, filed March 19, 2015, the disclosure of which is incorporated herein in its entirety. [Industrial Applicability]

[0136] INDUSTRIAL APPLICABILITY The present invention can be used in an object detection device that detects objects such as people and vehicles using a dictionary from a road image captured by a surveillance camera. [Explanation of symbols]

[0137] 1. Object detection device 2... Input image 3. Display device 4. Input device 10. Dictionary 11…Detection section 12. Reception 13...Generation section 14…Learning Department 100...Object detection device 101…Detection unit 102...Dictionary 103…Detection result DB 104…Reception 105...Generation section 106…Teacher data memory 107…Learning Department 111...Image 112, 113, 114, 115...Search window 120...Display device 121…Reception screen 122...Graph 123...Slider 123a…Slide bar 124...Image window 125...Rectangular frame 125a...partial area 126, 127...Reception screen 130…Input device 131…Object 132...Circumscribing rectangle

Claims

1. a display control means for displaying a first frame indicating an area in which an object is detected from an input image; a first receiving means for receiving an input of a second frame indicating an area including an object different from the detected object with respect to the input image in which the area indicating the first frame is displayed; a second receiving means for, when receiving an input of a first class indicating a type of object for the area indicated by the second frame, receiving an input of a second class indicating a type of object different from the first class for a partial area where the area indicated by the first frame and the area indicated by the second frame overlap; An information processing system comprising:

2. 2. The information processing system according to claim 1, further comprising a determination unit that determines whether an area indicated by said second frame overlaps with an area indicated by said first frame.

3. The first reception means receives an input for arranging the second frame from a user.

3. The information processing system according to claim 1 or 2.

4. displaying a first frame indicating an area in which an object is detected from the input image; receiving an input of a second frame indicating an area including an object different from the detected object with respect to the input image in which the area indicating the first frame is displayed; when receiving an input of a first class indicating a type of object for the area indicated by the second frame, receiving an input of a second class indicating a type of object different from the first class for a partial area where the area indicated by the first frame and the area indicated by the second frame overlap; Information processing methods.

5. The information processing method according to claim 4 , further comprising determining whether an area indicated by the second frame and an area indicated by the first frame overlap with each other.

6. receiving an input from a user to place the second frame; 6. The information processing method according to claim 4 or 5.

7. A process of displaying a first frame indicating an area in which an object is detected from an input image; a first process of receiving an input of a second frame indicating an area including an object different from the detected object with respect to the input image in which the area indicating the first frame is displayed; a second process of accepting, when an input of a first class indicating a type of object is accepted for the area indicated by the second frame, an input of a second class indicating a type of object different from the first class for a partial area where the area indicated by the first frame and the area indicated by the second frame overlap; program.

8. The program according to claim 7 , further causing the computer to execute a process of determining whether an area indicated by the second frame overlaps with an area indicated by the first frame.

9. In the first process, an input for arranging the second frame is received from a user. The program according to claim 7 or 8.

Images