Object detection device, method, and program

Abstract

In one embodiment of the present invention, when detecting an object from a first image having distortion, first, a wide-area second image in which the distortion has been corrected is generated on the basis of the first image, a first image area including an object designated as an object to be detected is detected from the generated second image, a first score corresponding to the name of the object in the first image area is obtained, and an image area in the first image area where the first score satisfies a first condition is selected as a candidate area. Next, a detailed third image including the candidate area is generated, a second image area including the object is detected from the generated third image, a second score corresponding to the name of the object in the second image area is obtained, and an image area in the second image area where the second score satisfies a second condition is selected as an image representing the object to be detected.

Description

Object detection device, method, and program

[0001] One aspect of the present invention relates to an object detection device, method, and program used to detect a desired object from an image captured by, for example, an omnidirectional camera.

[0002] A method called YOLO (You Only Look Once) is known as a method for detecting a desired object from an image. This method uses a neural network to detect a bounding rectangle of an object, calculates scores corresponding to the vertices of the bounding rectangle containing the object and the class name of the object from a given image, and extracts bounding rectangles with scores equal to or greater than a threshold to detect the desired object (see, for example, Non-Patent Document 1).

[0003] Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, “You Only Look Cnce: Unified, Real-time Object Detection”, arXiv:1506.02640v5 [cs.CV] 9 May 2016. Internet <URL: https: / / arxiv.org / abs / 1506.02640>

[0004] However, when applying the method described in Non-Patent Document 1 to detect an object from an equirectangular image captured by an omnidirectional camera, for example, the following problem occurs. That is, because an equirectangular image represents a wide range in all directions in a single image, the image becomes distorted into a horizontally elongated shape from the center toward the top and bottom edges. Therefore, when applying YOLO to detect a desired object from an equirectangular image, the distortion may result in missed detections or false detections.

[0005] The present invention has been made in light of the above circumstances, and aims to provide a technique that enables an object to be detected with high accuracy even from an image that has partial distortion.

[0006] In order to solve the above problem, one aspect of the object detection device or detection method of the present invention detects an object from a distorted first image by first generating a wide-area second image in which the distortion has been corrected based on the first image, detecting a first image region including an object designated as a detection target from the generated second image, calculating a first score corresponding to an object name of the first image region, and selecting, as a candidate region, an image region of the first image region whose first score satisfies a first condition. Next, a detailed third image including the candidate region is generated, detecting, from the generated third image, a second image region including the object, calculating a second score corresponding to the object name of the second image region, and selecting, as an image representing the object designated as a detection target, an image region of the second image region whose second score satisfies a second condition.

[0007] According to one aspect of the present invention, even if the acquired image is an image that has partial distortion, such as an equirectangular image, candidate areas containing objects are first searched for from a wide-area image in which the distortion has been corrected, making it possible to thoroughly search for candidate areas containing objects without being affected by the distortion of the equirectangular image.

[0008] In addition, a detailed image including the candidate area searched from the wide-area image is generated, and an image area including the object is searched from the generated detailed image. An image area among these image areas whose score satisfies the second condition is selected as the object detection result, thereby reducing false detections.

[0009] That is, according to one aspect of the present invention, it is possible to provide a technique that enables an object to be detected with high accuracy even from an image that has partial distortion.

[0010] FIG. 1 is a block diagram showing an example of the hardware configuration of an object detection device according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the software configuration of an object detection device according to an embodiment of the present invention. FIG. 3 is a flowchart showing an example of the processing procedure and processing content of an object detection process executed by a control unit of the object detection device shown in FIG. 2. FIG. 4 is a diagram illustrating a process for generating a wide-area image in which distortion of an equirectangular image has been corrected. FIG. 5 is a diagram illustrating an example of distortion correction characteristics. FIG. 6 is a diagram illustrating an example of a process for extracting a candidate area for an object using image search. FIG. 7A is a diagram illustrating a first example of setting a margin according to the size of a candidate area in a detailed image generation process. FIG. 7B is a diagram illustrating a second example of setting a margin according to the size of a candidate area in a detailed image generation process. FIG. 7C is a diagram illustrating a third example of setting a margin according to the size of a candidate area in a detailed image generation process. FIG. 8 is a diagram illustrating an example of a process from a process for detecting a candidate area in a distortion-corrected wide-area image to a process for determining a candidate area. FIG. 9 is a diagram illustrating an example of a process for detecting an object using the YOLO technique.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] [One Embodiment] (Configuration Example) FIGS. 1 and 2 are block diagrams showing an example of the hardware configuration and software configuration, respectively, of an object detection device CS according to one embodiment of the present invention.

[0013] The object detection device CS according to one embodiment is, for example, a personal computer, and includes a control unit 1 that uses a hardware processor such as a central processing unit (CPU). A storage unit having a program storage unit 2 and a data storage unit 3, and an input / output interface (hereinafter, interface will be abbreviated as I / F) unit 4 are connected to the control unit 1 via a bus 5. The object detection device CS may be provided in an edge computer or a server computer located on the web or in the cloud.

[0014] The camera CM, input device IN, and output device OU are connected via, for example, signal cables to the input / output I / F unit 4. Note that, instead of signal cables, a low-power wireless interface such as Bluetooth (registered trademark) may also be used as a means for connecting the input / output I / F unit 4 with the camera CM, input device IN, and output device OU.

[0015] The camera CM is, for example, a camera capable of capturing images in all directions, and outputs data representing an equirectangular image captured in all directions.

[0016] The input device IN is composed of, for example, a keyboard, a mouse, etc., and is used to input the name of an object designated by the user as a detection target, as well as data representing a candidate threshold and a judgment threshold for narrowing down the candidates.

[0017] The output device OU is, for example, a display device, and is used to display the object detection information generated by the control unit 1. Note that the output device OU may be a printer that prints the object detection information or an external memory that stores the object detection information.

[0018] Furthermore, by providing the input / output I / F unit 4 with a communication interface function, the object detection device CS can also transmit object detection information to other terminals, such as a mobile terminal owned by the user, via a communication network.

[0019] The program storage unit 2 is, for example, a combination of a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD) as a storage medium that can be written to and read from at any time, and a non-volatile memory such as a read only memory (ROM), and stores application programs necessary for executing various processes related to one embodiment of the present invention, in addition to middleware such as an operating system (OS).

[0020] The data storage unit 3 is configured by combining, for example, a nonvolatile memory such as an HDD or SSD as a storage medium that can be written to and read from at any time with a volatile memory such as a RAM (Random Access Memory). The data storage unit 3 is used to store data representing the equirectangular image output from the camera CM, input data indicating the name of the object to be detected and each threshold value for narrowing down the candidates input by the input device IN, various image data generated in the process of the object detection process, coordinate data indicating the candidate area, etc.

[0021] The control unit 1 includes, as processing function units according to one embodiment of the present invention, a data input reception processing unit 11, a candidate generation processing unit 12, a wide-area image search processing unit 13, a candidate determination processing unit 14, a detailed image search processing unit 15, and an object detection information output processing unit 16.

[0022] Each of the processing units 11 to 16 is realized by causing a hardware processor in the control unit 1 to execute an application program stored in the program storage unit 2. Note that some or all of the processing units 11 to 16 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0023] The data input reception processing unit 11 receives input data, which is input by the input device IN and indicates the name of the object to be detected and a threshold value for narrowing down the candidates, via the input / output I / F unit 4, and stores the data in the input data storage area of ​​the data storage unit 3. The data input reception processing unit 11 also receives data, which indicates an equirectangular image showing all directions and is output from the camera CM, via the input / output I / F unit 4, and stores the data in the image storage area of ​​the data storage unit 3.

[0024] The candidate generation processing unit 12 generates a wide-area image with distortion corrected based on the equirectangular image and stores the image in the image storage area of ​​the data storage unit 3. Furthermore, the candidate generation processing unit 12 selects, from each candidate area searched by the wide-area image search processing unit 13 (described later), a candidate area whose score is equal to or greater than a candidate threshold value input in advance.

[0025] In response to a search request from the candidate generation processing unit 12, the wide-area image search processing unit 13 searches the wide-area image for all candidate areas represented by circumscribing rectangles containing objects, and calculates a score corresponding to the object name for each searched candidate area.

[0026] The candidate determination processing unit 14 generates a detailed image for each candidate area generated by the candidate generation processing unit 12, and stores the generated detailed images in the image storage area of ​​the data storage unit 3. Furthermore, the candidate determination processing unit 14 selects, from image areas containing objects searched from the detailed images by the detailed image search processing unit 15 (described later), image areas whose scores satisfy a determination threshold input in advance, and stores the selected image areas in the data storage unit 3.

[0027] In response to a search request from the candidate determination processing unit 14, the detailed image search processing unit 15 searches for an image area consisting of a circumscribing rectangle containing an object from the detailed image, and calculates a score corresponding to the object name of the searched image area.

[0028] After the candidate determination process for all candidate areas is completed, the object detection information output processing unit 16 reads information about the object contained in the finally selected image area from the data storage unit 3, generates object detection information, and outputs the generated object detection information from the input / output I / F unit 4 to the output device OP.

[0029] (Example of Operation) Next, an example of operation of the object detection device CS configured as above will be described.

[0030] FIG. 3 is a flowchart showing an example of the processing procedure and processing content of the object detection processing executed by the control unit 1 of the object detection device CS.

[0031] (1) Acquisition of Input Data Prior to the object detection process, in step S10, the control unit 1 of the object detection device CS, under the control of the data input reception processing unit 11, acquires data representing the name of the object to be detected, input by the user via the input device IN, and data representing a candidate threshold Tc and a judgment threshold Tr for narrowing down the candidates, via the input / output I / F unit 4, and stores the acquired input data in the data storage unit 3.

[0032] (2) Acquisition of Omnidirectional Image Data Next, in step S11, the control unit 1 of the object detection device CS receives an omnidirectional image captured by the camera CM, such as equirectangular image data I, via the input / output I / F unit 4 under the control of the data input reception processing unit 11, and stores the received equirectangular image data I in the data storage unit 3.

[0033] (3) Candidate Generation Process When the equirectangular image data I is acquired, the control unit 1 of the object detection device CS, under the control of the candidate generation processing unit 12, executes the process of detecting a candidate area including an object as follows.

[0034] That is, first, in step S12, the candidate generation processing section 12 reads out the equirectangular image data I from the data storage section 3, and generates a wide area image J by correcting the distortion of the read out equirectangular image data I.

[0035] 4, the aspect ratio of an equirectangular image is close to 1 near the center of the image in the vertical direction, but the horizontal size increases from the center toward the top and bottom edges. In other words, the image has horizontally elongated distortion near the top and bottom edges.

[0036] Therefore, the candidate generation processing unit 12 corrects the image using a function that stretches the image size vertically from near the vertical center of the equirectangular image toward the top and bottom edges. Figure 5 shows an example of a correction function used to correct the horizontal distortion. As shown in Figure 5, the correction function is set so that the stretching rate y increases as the position in the x-axis direction, which indicates the vertical direction of the image, moves from the center (x=0) of the screen toward the top edge (x=1) and bottom edge (x=-1). By correcting the equirectangular image using such a correction function, a wide-area image J is generated in which the top and bottom edges of the image are stretched vertically, as shown in Figure 4, for example.

[0037] More specifically, the candidate generation processing unit 12 generates the wide-area image J as follows: If the color of a pixel at coordinates (x', y') is J(x', y'), the color of the pixel J(x', y') is expressed as follows:

[0038]

[0039] Here, adjust_equiv(x', y' | R) is expressed as follows:

[0040]

[0041] The adjust_equiv function is a function that converts a wide-area image into an equirectangular image, and in this example, corrects distortion of the equirectangular image using an inverse stretching function.

[0042] The candidate generation processing unit 12 then requests the wide-area image search processing unit 13 to search for candidate areas containing the object, using the wide-area image generated as described above and the object name stored in the data storage unit 3 as arguments.

[0043] Upon receiving the search request, the wide-area image search processing unit 13 searches the wide-area image for all circumscribing rectangles containing the object specified by the object name using the YOLO method in step S13, calculates a score representing the accuracy rate for each circumscribing rectangle, and then returns the coordinate data of each circumscribing rectangle and its score to the candidate generation processing unit 12.

[0044] 6 shows an example of image search processing by the wide-area image search processing unit 13. In this example, a circumscribing rectangle including "Japan" is searched for from a world map. Note that the YOLO method is well known as described in Non-Patent Document 1, and therefore a detailed description thereof will be omitted here.

[0045] Based on the coordinate data of each circumscribing rectangle and its score notified by the wide area image search processing unit 13, the candidate generation processing unit 12 deletes from the circumscribing rectangles those having scores less than the candidate threshold Tc stored in the data storage unit 3, selects circumscribing rectangles having scores equal to or greater than the candidate threshold Tc as candidate areas, and stores the coordinate data of those in the data storage unit 3.

[0046] (4) Candidate Determination Process When the candidate area generation process by the candidate generation processing unit 12 is completed, the control unit 1 of the object detection device CS then executes a determination process under the control of the candidate determination processing unit 14 to further narrow down the object candidates from among the multiple candidate areas as follows.

[0047] FIG. 8 is a diagram for explaining an example of the candidate determination process performed by the candidate determination processing unit 14. In FIG.

[0048] That is, the candidate determination processing unit 14 first selects one of the candidate areas in step S15. Then, in step S16, the candidate determination processing unit 14 converts the coordinate system of the wide-area image into the coordinate system of the equirectangular image, and then generates a detailed image including a circumscribing rectangle that is the selected candidate area.

[0049] For example, assume that the candidate region searched from the wide-area image is expressed by the following formula:

[0050]

[0051] In this case, the candidate determination processing unit 14 first converts the coordinate system of the wide-area image into the coordinate system of the equirectangular image as follows.

[0052]

[0053] The candidate determination processing unit 14 then generates a detailed image including the candidate region based on the equirectangular image.

[0054] At this time, if the angle of view of the circumscribing rectangle indicating the candidate area is small, it is possible that only a portion of the object is included in the circumscribing rectangle, or that the position of the circumscribing rectangle is shifted from the object. Therefore, as shown in Figure 7A, for example, the candidate determination processing unit 14 sets a large margin M1 so that the circumscribing rectangle E1 is reliably included, and the entire object is included in the detailed image.

[0055] On the other hand, if the angle of view of the circumscribing rectangle E3 of the candidate is large, setting a large margin would result in significant image distortion or unwanted objects being captured, so the candidate determination processing unit 14 sets the margin M3 to a small value, as shown in Fig. 7C, for example. Also, if the angle of view of the circumscribing rectangle E2 of the candidate is medium, the margin M2 is set to a medium value, as shown in Fig. 7B, for example.

[0056] The pixel value of the detailed image at the coordinates (u, v) of the equirectangular image is P l It is calculated as (u,v) = I(x,y).

[0057] The candidate determination processing unit 14 then l The coordinates (u,v) of the image are converted to the coordinates (x,y) of the equirectangular image as follows:

[0058]

[0059]

[0060] The candidate determination processing unit 14 then determines the object name and the detailed image P l The detailed image P l Requests a search for a bounding rectangle containing the object.

[0061] In step S17, the detailed image search processing unit 15 searches for the detailed image P l The system searches for a bounding rectangle containing an object corresponding to the object name from the list, and calculates a score representing the accuracy rate for the object name for the bounding rectangle that was found. The system then notifies the candidate determination processing unit 14 of the coordinate data of the bounding rectangle and its score in the following format:

[0062]

[0063] FIG. 9 shows a detailed image P l 1 shows an example of a bounding rectangle searched from the image and its score.

[0064] In step S18, the candidate determination processing unit 14 first selects the detailed image P l It is determined whether or not there are multiple bounding rectangles found from the search results. If multiple bounding rectangles are found, only the bounding rectangle with the highest score is selected from these bounding rectangles.

[0065]

[0066] The candidate determination processing unit 14 then converts the coordinates of the selected circumscribing rectangle into the coordinate system of the equirectangular image as follows.

[0067]

[0068] The candidate determination processing unit 14 further deletes circumscribing rectangles with scores less than a determination threshold Tr from the above circumscribing rectangles, and stores circumscribing rectangles with scores equal to or greater than the threshold Tr as object detection results in the data storage unit 3. The following formula shows the object detection results.

[0069]

[0070] When the candidate determination processing unit 14 has completed the above determination process for one candidate area, it determines in step S19 whether or not all of the multiple candidate areas searched for in the wide-area image have been selected. If any unselected candidate areas remain, the candidate determination processing unit 14 returns to step S15 to select the next candidate area and executes the aforementioned series of candidate determination processes in steps S16 to S18. Thereafter, the candidate determination processing unit 14 repeatedly executes the candidate determination process for each remaining candidate area in the same manner. When the processing for all candidate areas has been completed, the candidate determination processing ends.

[0071] (5) Output of Object Detection Information Finally, in step S20, the control unit 1 of the object detection device CS, under the control of the object detection information output processing unit 16, reads information representing the object detection results stored in the data storage unit 3 and generates display data representing the read information representing the object detection results. The generated display data is then output from the input / output I / F unit 4 to the output device OU. Thus, information representing the desired object detected from the omnidirectional image captured by the camera CM is displayed on the output device OU.

[0072] (Effects) As described above, in one embodiment, when detecting an object from an equirectangular image captured by a camera CM, a wide-area image is first generated by correcting distortion from the equirectangular image, and a circumscribing rectangle containing the object is searched for in the generated wide-area image and its score is calculated. From the searched circumscribing rectangles, a circumscribing rectangle with a score equal to or greater than a candidate threshold Tc is selected, and the selected circumscribing rectangle is designated as a candidate area. Next, a detailed image of the range containing the candidate area is generated, and a circumscribing rectangle containing the object is searched for in this detailed image and its score is calculated. From the searched circumscribing rectangles, a circumscribing rectangle with a score equal to or greater than a determination threshold Tr and with the highest score is selected, and the selected circumscribing rectangle is designated as a region containing the object to be detected, and its detection information is output.

[0073] Therefore, even if the acquired image is an equirectangular image, a circumscribing rectangle including the object is first searched for from the wide-area image in which the distortion has been corrected, so that it is possible to search for a circumscribing rectangle including the object without omission, without being affected by the distortion of the equirectangular image.

[0074] Furthermore, a detailed image is generated that includes the circumscribing rectangle searched from the wide-area image, and a circumscribing rectangle containing the object is searched for from the generated detailed image. From among these circumscribing rectangles, the circumscribing rectangle with the highest score that is equal to or greater than the judgment threshold is selected as the object detection result, thereby reducing false detections.

[0075] That is, even from an equirectangular image, it is possible to detect an object with high accuracy by reducing the occurrence of missed detections and false detections.

[0076] Other Embodiments (1) In one embodiment, a stretching function used to correct distortion in an equirectangular image is sin -1 However, the present invention is not limited to this function. For example, any function can be used as long as the enlargement ratio moves in a direction larger than y=x (no correction) as the vertical position of the image approaches x=1, and the enlargement ratio moves in a direction smaller than y=x as the vertical position of the image approaches x=−1.

[0077] (2) In addition, the functional configuration, processing procedure, processing content, types of distorted images, types of objects to be detected, etc. of the object detection device can be modified and implemented in various ways without departing from the spirit of this invention.

[0078] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0079] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0080] CS...Object detection device CM...Camera IN...Input device OU...Output device 1...Control unit 2...Program storage unit 3...Data storage unit 4...Input / output I / F unit 5...Bus 11...Data input reception processing unit 12...Candidate generation processing unit 13...Wide area image search processing unit 14...Candidate determination processing unit 15...Detailed image search processing unit 16...Object detection information output processing unit

Claims

1. An object detection device comprising: a first processing unit that acquires a first image having distortion; a second processing unit that generates a wide-area second image based on the first image with the distortion corrected; a third processing unit that detects a first image area from the second image that includes an object designated as a detection target, and calculates a first score corresponding to the object name of the first image area; a fourth processing unit that selects, as a candidate area, an image area from the first image area where the first score satisfies a first condition; a fifth processing unit that generates a detailed third image that includes the candidate area, detects a second image area from the generated third image that includes the object, and calculates a second score corresponding to the object name of the second image area; and a sixth processing unit that selects, as an image representing the object to be detected, an image area from the second image area where the second score satisfies a second condition.

2. The object detection device according to claim 1, wherein when the first image is an equirectangular image, the second processing unit performs the correction using a correction function set to stretch the image vertically from the center of the image toward the top and bottom edges of the image.

3. The object detection device according to claim 1, wherein the fifth processing unit generates the third image corresponding to an area including a margin set according to the size of the candidate area.

4. The object detection device according to claim 1, wherein the sixth processing unit selects, from the second image region, an image region in which the second score is equal to or greater than a judgment threshold preset as the second condition and in which the second score is the maximum, as an image representing the object to be detected.

5. An object detection method executed by an information processing device, comprising the steps of: acquiring a first image having distortion; generating a wide-area second image in which the distortion has been corrected based on the first image; detecting a first image area including an object designated as a detection target from the second image and calculating a first score corresponding to the object name of the first image area; selecting, as a candidate area, an image area from the first image area whose first score satisfies a first condition; generating a detailed third image including the candidate area, detecting a second image area including the object from the generated third image and calculating a second score corresponding to the object name of the second image area; and selecting, as an image representing the object to be detected, an image area from the second image area whose second score satisfies a second condition.

6. A program that causes a processor included in an object detection device to execute the processing performed by at least one of the first to sixth processing units included in the object detection device according to any one of claims 1 to 4.

Images