OBJECT DETECTION DEVICE, OBJECT DETECTION METHOD, AND PROGRAM
The two-stage object detection system optimizes processing time and accuracy by using a higher accuracy second stage and excluding similar candidate regions based on previous false detections, ensuring efficient and precise object detection.
Patent Information
- Application Number
- JP2021036637
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-08
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2041-03-08
AI Technical Summary
Conventional two-stage object detection methods suffer from increased processing time due to false detections in the first stage, leading to redundant processing in the second stage, especially with fixed cameras where non-target objects remain unchanged.
A two-stage object detection system where the first stage identifies candidate regions and the second stage uses a higher accuracy algorithm, with a determining mechanism to reduce redundant processing by excluding candidate regions similar to previous false detections, using detection information from previous frames to determine target areas.
This approach enables high-speed and precise object detection by reducing unnecessary processing, maintaining detection performance while minimizing repetitive false detections.
Smart Images

Figure 0007581980000001 
Figure 0007581980000002 
Figure 0007581980000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for detecting an object. [Background technology]
[0002] Conventionally, there has been known a technology for detecting an object using a two-stage detector divided into a front stage and a rear stage. For example, in Patent Document 1 and Patent Document 2, a front stage detector detects candidate regions of a detection target (e.g., a face), and a rear stage detector detects the detection target from the multiple candidate regions, thereby performing highly accurate object detection. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2006-293720 A [Patent Document 2] JP 2019-021001 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, in the conventional technology, there is a problem that the processing time increases by performing two-stage detection processing. In particular, with a fixed camera, the appearance does not change except for the object to be detected (e.g., a moving object), so a false detection that occurs once in a detector in the previous stage occurs repeatedly in the same place. The detection process is performed again in the detector in the later stage even in the falsely detected area, which causes a problem of further increasing the processing time.
[0005] An object of the present invention is to provide a technique capable of detecting an object quickly and accurately. [Means for solving the problem]
[0006] In order to achieve the above object, the present invention employs the following configuration.
[0007] A first aspect of the present invention is an object detection device that detects a predetermined object from an image, comprising: a first detection means that detects a candidate area in which the object exists from the image; a determination means that determines a target area from one or more of the candidate areas detected by the first detection means; a second detection means that detects the object using a detection algorithm different from that used by the first detection means, with the target area as the target; and a storage means that stores detection information representing the detection result by the second detection means for the target area, wherein the determination means determines the target area from one or more of the candidate areas based on the detection information for one or more previous frames.
[0008] The object to be detected is not particularly limited, but may be, for example, a human body, a face, a specific animal, a car, or a specific product. The candidate area is an area determined by the first detection means to have a high probability of containing the object to be detected, and the area to be detected by the second detection means (target area) is determined based on this candidate area. The first detection means and the second detection means may use any algorithm, but it is preferable that the detection algorithm of the second detection means is capable of detection with higher accuracy and requires more calculations than the detection algorithm of the first detection means. The detection information is information obtained by performing an object detection process performed by the second detection means, and includes, for example, the position and size of the target area, an image corresponding to the target area, and a score indicating the likelihood that the object to be detected is contained in the target area.
[0009] The detection information includes information about a target area where an object was not detected by the second detection means. It is preferable that the second detection means includes a first detection reliability indicating the likelihood that an object is included in the candidate area. In this case, the determination means may determine, as the target area, the candidate area other than the candidate area having a similarity to the target area in which no object was detected in the previous frame that is equal to or greater than a predetermined value. The first detection means may also output a first detection reliability indicating the likelihood that the candidate area includes an object, and the determination means may determine the target area based on a value obtained by subtracting a predetermined value from the first detection reliability for the candidate area having a similarity to the target area in which no object was detected in the previous frame that is equal to or greater than a predetermined value, and based on the first detection reliability for the other candidate areas. According to the above configuration, the number of candidate areas passed to the second detection means is reduced, so that the processing time can be reduced while maintaining the detection performance by performing a two-stage detection process.
[0010] The predetermined value to be subtracted from the first detection reliability may be a value corresponding to the number of consecutive frames in which the second detection means did not detect an object. For example, the predetermined value may be increased as the number of consecutive frames increases, or the predetermined value to be subtracted from the first detection reliability may be subtracted only when the number of consecutive frames is equal to or greater than a certain number. The predetermined value to be subtracted from the first detection reliability may be a fixed value.
[0011] The first detection means also outputs a first detection reliability indicating the likelihood that an object is included in the candidate area, and the detection information includes a second detection reliability indicating the likelihood that an object is included in the target area determined by the second detection means, and the determination means determines the target area based on a value obtained by subtracting a value corresponding to the second detection reliability from the first detection reliability for candidate areas whose similarity to the target area indicated in the detection information is equal to or greater than a predetermined value, and based on the first detection reliability for other candidate areas. For example, the higher the second detection reliability, the larger the predetermined value to be subtracted from the first detection reliability may be.
[0012] The detection information may include the position and / or size of the target area, and the determination means may obtain the similarity based on the position and / or size of the candidate area and the position and / or size of the target area. In object detection, false detection may occur multiple times for the same object in the input image, but the above configuration can effectively reduce false detection of the same object at the same position and size multiple times. This reduces the number of candidate areas across the second detection unit, and by performing two-stage detection processing, it is possible to reduce processing time while maintaining detection performance.
[0013] The detection information may include an image corresponding to the target region, and the determination means may obtain a similarity based on the image included in the detection information and the image corresponding to the candidate region. This allows for highly accurate object detection even when the position or size of the region corresponding to the false detection information and the candidate region match or are similar, but the images corresponding to the two regions are completely different.
[0014] A second aspect of the present invention is an object detection method for detecting a predetermined object from an image, comprising: a first detection step of detecting a candidate area in which the object exists from the image; a determination step of determining a target area from one or more of the candidate areas detected in the first detection step; a second detection step of detecting the object in the target area using a detection algorithm different from that used in the first detection step; and a storage step of storing detection information representing the detection result in the second detection step for the target area, wherein in the determination step, the target area is determined from one or more of the candidate areas based on the detection information for one or more previous frames.
[0015] The present invention may be understood as an object detection device having at least a part of the above means, or as a device for recognizing or tracking a detection target object, or as an image processing device or a surveillance system. The present invention may also be understood as an object detection method, object recognition method, object tracking method, image processing method, or surveillance method including at least a part of the above processing. The method may be realized as a program for implementing the method, or as a recording medium on which the program is non-temporarily recorded. Note that the above-mentioned means and processes may be combined with each other as much as possible to constitute the present invention. Effect of the Invention
[0016] According to the present invention, object detection can be performed quickly and accurately. [Brief description of the drawings]
[0017] [Figure 1] FIG. 1 is a diagram illustrating an application example of object detection. [Diagram 2] FIG. 2 is a diagram illustrating a configuration of the object detection device. [Diagram 3] FIG. 3 is a flowchart of the object detection process. [Figure 4] FIG. 4 is a flowchart of the determination process. [Diagram 5] FIG. 5 is a flowchart of the determination process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0018] (Examples) With reference to FIG. 1, an application example of the object detection device according to the present invention will be described. The object detection device detects an object (for example, a human body) from an image acquired by a fixed camera attached above an area to be detected (for example, a ceiling). The object detection device uses a two-stage detector divided into a front stage and a rear stage. Objects 101 and 102 are detected objects (for example, a human body) and are moving objects moving within the imaging range of the fixed camera 1. Object 103 is an object (for example, a flower) provided within the imaging range of the fixed camera 1. The object detection device detects candidate areas 111 to 113 in which objects exist using the above-mentioned front stage detector for an input image. The candidate areas 111 to 113 are areas corresponding to the objects 101 to 103. Although the object 103 is not the human body to be detected, the candidate area 113 occurs when the characteristics of the object 103 are similar to a human body. Then, the object detection device performs object detection using the above-mentioned rear stage detector, and records the detection result in a storage device. The latter detector basically targets object regions 121-123 corresponding to the candidate regions 111-113. Here, it is assumed that the former detector erroneously detects the object (flower) 103 as the object, but the latter detector can detect it as not being the object. In this case, it is considered that the former detector will continue to erroneously detect the object 103. If all the candidate regions are treated as the object regions of the latter detector, in the situation of FIG. 1, the latter detector will perform detection processing for every frame even though there is no object, resulting in unnecessary processing.
[0019] Therefore, in this application example, a region (target region) in which a later detector performs object detection is determined from among regions (candidate regions) in which a previous detector detects an object, based on detection information for one or more previous frames. For example, a candidate region in a current frame that is highly similar to a region in which a later detector did not detect an object in one or more previous frames may be excluded from the target region. Alternatively, a target region is determined from among the candidate regions based on the detection score (reliability) of a previous detector, but a region in which a later detector did not detect an object in one or more previous frames may be determined as a target region based on a value obtained by subtracting a predetermined value from the detection score. The value to be subtracted may be a fixed value or a value according to the number of consecutive frames in which an object was not detected. In this way, even if a region in which a previous detector detected an object is similar to a region in which a later detector did not detect an object, the region can be excluded from the processing target of the later detector, thereby speeding up processing while maintaining the accuracy of object detection.
[0020] (Embodiment 1) <Configuration> 2 is a functional block diagram of the object detection device 10 according to the present embodiment. The device 10 is an information processing device (computer) including a calculation device (CPU; processor), a memory, a storage device (storage unit 16), an input / output device, etc. The object detection device 10 executes a program stored in the storage device to provide functions such as an image input unit 11, a first detection unit 12, a determination unit 13, a second detection unit 14, and an output unit 15. Some or all of these functions may be implemented by a dedicated logic circuit such as an ASIC or an FPGA.
[0021] The image input unit 11 has a function of importing image data from the camera 20. The imported image data is passed to the first detection unit 12. This image data may be stored in the storage unit 16. In this embodiment, the image data is received directly from the camera 20, but the image data may be received via a communication device or the like, or via a recording medium. The image to be input is not particularly limited, and may be an RGB image, a gray image, or an image representing distance or temperature, etc.
[0022] The first detection unit 12 detects a candidate region (a region where a detection target object is likely to exist) from an input image. In this embodiment, the first detection unit 12 detects the candidate region using a detector using Haar-like features and adaboost. The detection result is handed over to the determination unit 13. The detection result includes the detected candidate region, and may further include the likelihood that the detection target object exists in the candidate region (first detection reliability, detection score). Note that the features used for detection and the learning algorithm of the detector are not particularly limited. For example, any feature such as a Histgram of Gradient (HoG) feature, a SIFT feature, a SURF feature, or a Sparse feature can be used as the feature. In addition, any learning method such as a boosting method other than adaboost, a Support Vector Machine (SVM), a neural network, or a decision tree learning can be used as the learning algorithm.
[0023] The determination unit 13 determines an area (target area) to be detected by the second detection unit 14 from among the candidate areas detected by the first detection unit 12. In this embodiment, the determination unit 13 determines the target area from among the candidate areas using detection information of the previous frame stored in the storage unit 16. The detection information includes information on a target area (falsely detected area) in which an object was not detected by the second detection unit 14 described later in one or more previous frames. The determination unit 13 determines, as the target area, the candidate areas other than the candidate areas having a similarity to the false detection area of a predetermined value or more, and outputs the candidate areas to the second detection unit 14 at the subsequent stage. Note that, when the detection result of the first detection unit 12 includes the above-mentioned first detection reliability, the determination unit 13 may determine, as the candidate area, an area excluding the candidate area similar to the false detection area from among the candidate areas having the first detection reliability of a predetermined value or more.
[0024] The second detection unit 14 performs object detection on the target area determined by the determination unit 13. The detection result includes information indicating whether or not the object to be detected exists in the target area, and may further include the likelihood that the object to be detected exists in the target area (second detection reliability, detection score), etc. In addition, in this embodiment, the second detection unit 14 records the position and / or size of the target area in which it is determined that the object to be detected does not exist as a result of the object detection, as detection information in the storage unit 16. Note that the second detection unit 14 may record the detection information (position and / or size) of all the target areas determined by the determination unit 13 in the storage unit 16. In this embodiment, the second detection unit 14 performs object detection using a detector using deep learning. Note that the method of deep learning is not particularly limited, and may be a detector using any method such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), SAE (Stacked Auto Encoder), DBN (Deep Belief Network), etc. In addition, the second detection unit 14 does not have to be a detector using deep learning. However, the detection algorithm of the second detection unit 14 may be the same as the detection algorithm of the first detection unit 12. It is desirable to have a method that can detect with higher accuracy and requires more calculations than the conventional method.
[0025] The output unit 15 outputs a detection result for an object detected by the second detection unit 14. For example, the output unit 15 outputs result information indicating that an object has been detected for a candidate area in which the reliability of the detection result by the second detection unit 14 is equal to or higher than a threshold. Candidate areas in which the reliability is less than the threshold need not be included in the result information. The detection result information is not particularly limited, but in the case of face detection, for example, information such as a face area, reliability, face direction, age, sex, race, and facial expression may be included.
[0026] <Processing content> 3 is a flowchart showing the overall flow of the object detection process by the object detection device 10. Details of the object detection device 100 will be described below with reference to the flowchart in FIG.
[0027] <S31: Image input processing> In step S31, the object detection device 10 acquires an image (input image). The input image may be acquired from the camera 20 via the image input unit 11, may be acquired from another computer via the communication device 104, or may be acquired from the storage unit 16.
[0028] <S32: First detection process> In step S32, the first detection unit 12 detects a candidate region (a region where a detection target object is estimated to exist) from the input image (first detection process). In this embodiment, the first detection unit 12 is configured to use Haar-like features as image features and AdaBoost as a learning algorithm. In addition to the above-mentioned candidate regions, the detection result of the first detection process may include the likelihood that the detection target object exists in the candidate region (first detection reliability, detection score).
[0029] <S33: Judgment process> In step S33, the determination unit 13 determines, as the target region, the candidate regions detected in step S32, other than the candidate regions having a similarity to the false detection region equal to or greater than a predetermined value. The false detection region is a target region in which no object was detected in the second detection process described later in the previous frame or frames. The determination unit 13 outputs, as the target region, the candidate regions detected in step S32, excluding those similar to the false detection region.
[0030] The determination process performed in step S33 will be described in detail with reference to FIG. 4. FIG. 4 is a flowchart of the determination process according to this embodiment. First, the determination unit 13 acquires detection information (the position and size of the false detection region) from the storage unit 16 (S41). The determination unit 13 may acquire only the false detection information for the immediately preceding frame, or may acquire the false detection information for a predetermined number of frames immediately preceding. Then, the determination unit 13 calculates the similarity between the false detection region and each of one or more candidate regions (S42). In this embodiment, IoU (Intersection over Union) is used as an index of similarity between regions. IoU is a value obtained by dividing the area of the intersection of two regions by the area of the union of the two regions. IoU takes a value between 0 and 1, and is 1 when the two regions completely overlap, and is 0 when they do not overlap at all. The position and size of the candidate region and the position and size of the false detection region may be used to calculate IoU. Then, the determination unit 13 determines whether or not the IoU is equal to or greater than a predetermined threshold T1 (S43), and if the IoU is equal to or greater than the threshold T1, outputs the region excluding the corresponding candidate region as a target region (S44).
[0031] <S34 to S36: Second detection process> In step S34, the second detection unit 14 detects one or more of the signals output in step S33. It is determined whether or not the object to be detected is included in the target region (second detection process). In this embodiment, the second detection unit 14 performs object detection using a classifier trained using a multi-layer neural network called a convolutional neural network (CNN).
[0032] In step S35, the second detection section 14 determines whether or not there is a target region that has been determined in the process of step S34 to not include the object to be detected.
[0033] In step S36, the second detection unit 14 records information about the target area determined not to include the object to be detected as detection information in the storage unit 16. In this embodiment, the position and size of the target area determined not to include the object to be detected are recorded in the storage unit 16 as the detection information.
[0034] <S37: Detection result output process> In step S37, output unit 15 outputs the detection result for the region in which the object was detected in step S34. Output unit 15 outputs result information indicating that the object to be detected has been detected for the detection target region in which the reliability of the detection result by the object detection region (second detection reliability) is equal to or greater than a threshold. Detection target regions in which the reliability is less than the threshold need not be included in the result information.
[0035] <Advantageous Effects of the Present Embodiment> In object detection, false positives may occur multiple times for the same object in the input image, but this embodiment can effectively reduce false positives of objects of the same position and size. This reduces the number of candidate regions (target regions) across the second detection unit, so that by performing two-stage detection processing, it is possible to reduce processing time while maintaining detection performance.
[0036] (Embodiment 2) In the above-mentioned first embodiment, an example was described in which the similarity is determined in step S33 based on the positions and sizes of the candidate region and the false detection region. In this embodiment, an example is described in which the similarity is determined in step S33 by performing pattern matching between an image corresponding to the candidate region and an image corresponding to the false detection region. Descriptions of the same processes as in the above-mentioned first embodiment will be omitted, and only the determination process (S33), which is a different process, will be described.
[0037] <Determination process (S33)> 5 is a flowchart of the determination process performed in step S33 in this embodiment. First, the determination unit 13 acquires detection information from the storage unit 16 (S51). In this embodiment, the detection information includes an image corresponding to a false detection region. Then, the determination unit 13 performs a pattern matching process for each of the images corresponding to one or more candidate regions using the image corresponding to the false detection region (S52). Then, the determination unit 13 determines whether the similarity between the images obtained by pattern matching is equal to or greater than a predetermined threshold T2 (S53), and if the similarity is equal to or greater than the threshold T2, outputs the region excluding the corresponding candidate region as the target region (S54).
[0038] <Advantageous Effects of the Present Embodiment> According to this embodiment, even if the positions and sizes of the false detection area and the candidate area are the same or similar, but the images corresponding to the two areas are completely different, it is possible to perform object detection with high accuracy. For example, even if the object to be detected overlaps the position of the object 103 shown in Fig. 1, the similarity is calculated based on the image, so that the area corresponding to that position can be set as the target area.
[0039] (Modification) In the above-mentioned first and second embodiments, the determination unit 13 has been described as determining candidate regions, excluding those similar to the false detection region, as the target region, as an example, but is not limited thereto. For example, when the first detection unit 12 outputs the above-mentioned first detection reliability, the determination unit 13 determines candidate regions, the first detection reliability of which is equal to or greater than a predetermined threshold T3, as the target region. At this time, for candidate regions, the determination unit 13 may determine candidate regions, the value of which obtained by subtracting a predetermined value from the first detection reliability is equal to or greater than the above-mentioned predetermined threshold T3, as the target region, for candidate regions, the similarity to the false detection region of which is equal to or greater than a predetermined threshold T4.
[0040] The method of determining the predetermined value to be subtracted from the first detection reliability is not particularly limited. The predetermined value to be subtracted from the reliability may be a fixed value. The predetermined value to be subtracted from the reliability may be determined according to the number of consecutive frames in which the second detection unit 14 did not detect the target object. For example, the predetermined value may be increased as the number of consecutive frames increases, or the predetermined value to be subtracted from the first detection reliability may be subtracted only when the number of consecutive frames is equal to or greater than a certain number. Furthermore, when the second detection unit 14 outputs the second detection reliability, the predetermined value to be subtracted from the reliability may be determined based on the second detection reliability. For example, the determination unit 13 determines a candidate region in which the first detection reliability is equal to or greater than a predetermined threshold T3 as the target region. At this time, for a candidate region in which the similarity to the erroneous detection region is equal to or greater than a predetermined threshold T4, the determination unit 13 may determine a candidate region in which the value obtained by subtracting a value based on the second detection reliability from the first detection reliability is equal to or greater than the above-mentioned predetermined threshold T3 as the target region. For example, the higher the second detection reliability, the larger the predetermined value to be subtracted from the reliability may be.
[0041] In the above-mentioned embodiment 1, an example of using IoU as an index of similarity between regions has been described, but it is not limited to this. For example, the ratio or difference in size between regions, the difference in position between regions (for example, center coordinate values), or a combination of these may be used as an index of similarity.
[0042] In the above-described second embodiment, an example of performing pattern matching when calculating the similarity between images has been described, but the present invention is not limited to this. For example, a difference in color information or a difference in luminance information between images may be used as an index of similarity. [Explanation of symbols]
[0043] 10: Object detection device 11: Image input section 12: First detector 13: Judgment section 14: Second detector 15: Output section 16: Storage part 1,20:Camera 101,102,103:Object 111,112,113: Candidate area 121, 122, 123:Target area
Claims
1. An object detection device for detecting a predetermined object from an image, a first detection means for detecting a candidate region in which the object exists from the image and outputting a first detection reliability indicating a likelihood that the object is included in the candidate region; a determination means for determining a target region from the one or more candidate regions detected by the first detection means; a second detection means for detecting the object using a detection algorithm different from that of the first detection means, the second detection means being configured to detect the object using the target area; a storage means for storing detection information representing a detection result by the second detection means for the target region, the detection information including information on a target region in which the object was not detected by the second detection means; having the determination means determines the target region based on a value obtained by subtracting a predetermined value from the first detection reliability for a candidate region having a similarity to a target region in which the object was not detected in the previous frame that is a predetermined value or more, and determines the target region based on the first detection reliability for other candidate regions.
1. An object detection device comprising:
2. the predetermined value is a value corresponding to the number of consecutive frames in which the object is not detected by the second detection means. The object detection device according to claim 1 .
3. The predetermined value is a fixed value. The object detection device according to claim 1 .
4. the detection information includes a position and / or a size of the region of interest; the determining means determines the degree of similarity based on a position and / or a size of the candidate region and a position and / or a size of the target region; 4. The object detection device according to claim 1, wherein the object detection device detects an object that is detected by the object detection unit.
5. the detection information includes an image corresponding to the region of interest; the determining means determines the degree of similarity based on the image included in the detection information and an image corresponding to the candidate region.
4. The object detection device according to claim 1, wherein the object detection device detects an object that is detected by the object detection unit.
6. An object detection method for detecting a predetermined object from an image, comprising: a first detection step of detecting a candidate region in which the object exists from the image and outputting a first detection reliability indicating a likelihood that the object is included in the candidate region; a determining step of determining a target region from one or more of the candidate regions detected in the first detection step; a second detection step of detecting the object using a detection algorithm different from that of the first detection step in the target region; a storage step of storing detection information representing a detection result in the second detection step for the target region, the detection information including information on a target region in which the object was not detected in the second detection step; having In the determining step, for a candidate region having a similarity to a target region in which the object was not detected in a previous frame equal to or greater than a predetermined value, the target region is determined based on a value obtained by subtracting a predetermined value from the first detection reliability, and for other candidate regions, the target region is determined based on the first detection reliability.
1. An object detection method comprising:
7. A program for causing a computer to execute each step of the object detection method according to claim 6.
Citation Information
Patent Citations
Face detection apparatus, face detection method, and face detection program
JP2006293720A
Object detection device, object detection method and program
JP2019021001A
Object detection device, object detection method, and program
JP2019159391A
