Human detection device, human detection method, and computer program product for human detection

Abstract

The present disclosure relates to a person detection device, a method, and a computer program product. The person detection device has: a detection section (31) that detects one or more person regions from an image; a consolidation section (33) that consolidates two or more person regions into one consolidated person region in a case where the two or more person regions overlap each other by a predetermined amount or more; a skeleton detection section (34) that determines a skeleton of an undetermined and foremost person among one or more persons included in the consolidated person region by inputting the consolidated person region to a skeleton detector; a mask section (35) that updates the consolidated person region by masking a region showing the foremost person within the consolidated person region based on the skeleton of the foremost person; and a repetition control section (36) that repeats the processing of the skeleton detection section and the mask section for the updated consolidated person region until a skeleton of a person is not detected, and counts the number of skeletons detected from the consolidated person region as the number of persons represented in the consolidated person region.

Description

TECHNICAL FIELD

[0001] The present application relates to a person detection device, a person detection method, and a computer program product for person detection that detect a person represented in an image. BACKGROUND

[0002] Techniques for detecting a person represented in an image obtained by a camera are being researched (see Japanese Patent Application Publication No. 2020-98474, Japanese Patent Application Publication No. 2018-22340, Japanese Patent Application Publication No. 2016-95808, and Japanese Patent Application Publication No. 2015-222881).

[0003] The attribute determination device disclosed in Japanese Patent Application Publication No. 2020-98474 detects the positions of control points that are points of a skeleton of a person from an image obtained by photographing a person from above, and identifies an attribute of the person using the detection result.

[0004] In addition, the image processing device disclosed in Japanese Patent Application Publication No. 2018-22340 estimates the number of specific objects in a first region of an image by performing a specific object detection process on the first region. Further, the image processing device estimates the number of specific objects in a second region of the image by performing a regression process that estimates the number of specific objects per predetermined region on the second region. Then the image processing device integrates the estimation results.

[0005] Further, the person detection device disclosed in Japanese Patent Application Publication No. 2016-95808 determines whether to integrate recognition results of a plurality of partial regions that are recognized as containing a predetermined object and that overlap each other, among a plurality of partial regions extracted from an image, based on distances in a depth direction of the partial regions.

[0006] Further, the monitoring device disclosed in Japanese Patent Application Publication No. 2015-222881 detects a person from a photographed image to acquire position information about a person region, and based on the position information, determines a region state that represents a state of the person in the person region. Also, the monitoring device sets a mask image corresponding to the region state, generates and outputs an output image obtained by changing the person region to the mask image corresponding to the region state. Further, the monitoring device acquires position information of each person in each frame, and in a case where a person fails to be detected in a frame in which a plurality of persons overlap, acquires position information in the frame from position information acquired in a plurality of frames immediately preceding the frame. SUMMARY

[0007] In the above-described techniques, in a case where a plurality of persons who are detection targets appear to overlap on an image, detection of each person sometimes fails.

[0008] Thus, an object of the present application is to provide a person detection device capable of detecting each person even when a plurality of persons overlap on an image.

[0009] According to one embodiment, a person detection device is provided. The person detection device has: a detection section that detects one or more person regions representing persons from an image generated by an imaging section; a consolidation section that, in a case where two or more person regions overlap by a threshold value or more, sets one consolidated person region by selecting one of the two or more person regions or by including the two or more person regions; a skeleton detection section that detects a skeleton of an undetermined and foremost person among one or more persons included in the consolidated person region by inputting the consolidated person region to a skeleton detector that is previously learned (trained) to detect a skeleton of a person; a mask section that updates the consolidated person region by masking a region representing the foremost person within the consolidated person region based on the skeleton of the foremost person; and a repetition control section that, for the updated consolidated person region, repeats processing by the skeleton detection section and the mask section until a skeleton of a person is not detected, and counts a number of skeletons detected from within the consolidated person region as a number of persons represented in the consolidated person region.

[0010] The person detection device preferably further has an overlap estimation section that estimates an overlap estimation region representing two or more persons on the image by tracking persons detected from each of a series of past images generated by the imaging section in time series before the image. In this case, the consolidation section preferably determines that each of the two or more person regions represents the same person in a case where the two or more person regions overlap by the threshold value or more and deviate from the overlap estimation region.

[0011] Further, the consolidation section preferably selects any one of the two or more person regions and deletes the other person regions in a case where the two or more person regions overlap by the threshold value or more and deviate from the overlap estimation region, and sets the consolidated person region to include a region that is a union of the two or more person regions in a case where the two or more person regions overlap by the threshold value or more and are included in the overlap estimation region.

[0012] In addition, the person detection device preferably further has an overlap estimation section that estimates an overlap estimation region representing two or more persons on the image by tracking persons detected from each of a series of past images generated by the imaging section in time series before the image. In this case, the consolidation section preferably sets the threshold value to be lower in a case where the two or more person regions are included in the overlap estimation region than in a case where the two or more person regions deviate from the overlap estimation region.

[0013] According to another embodiment, a person detection method is provided. The person detection method includes: detecting one or more person regions indicating a person from an image generated by an imaging section; in a case where the degree of overlap of two or more person regions is equal to or greater than a predetermined threshold, setting an integrated person region by selecting one of the two or more person regions or by including the two or more person regions; detecting a skeleton of an undetermined and foremost person among one or more persons included in the integrated person region by inputting the integrated person region to a skeleton detector that is previously learned to detect a skeleton of a person; updating the integrated person region by masking a region indicating the foremost person within the integrated person region based on the skeleton of the foremost person; and repeating the detection of the skeleton and the masking of the region indicating the person within the updated integrated person region until a skeleton of a person is not detected, and counting the number of skeletons detected from within the integrated person region as the number of persons indicated in the integrated person region.

[0014] According to still another embodiment, a person detection computer program product including a person detection computer program is provided. The person detection computer program has commands that cause a computer to execute the following processing: detecting one or more person regions indicating a person from an image generated by an imaging section; in a case where the degree of overlap of two or more person regions is equal to or greater than a predetermined threshold, setting an integrated person region by selecting one of the two or more person regions or by including the two or more person regions; detecting a skeleton of an undetermined and foremost person among one or more persons included in the integrated person region by inputting the integrated person region to a skeleton detector that is previously learned to detect a skeleton of a person; updating the integrated person region by masking a region indicating the foremost person within the integrated person region based on the skeleton of the foremost person; and repeating the detection of the skeleton and the masking of the region indicating the person within the updated integrated person region until a skeleton of a person is not detected, and counting the number of skeletons detected from within the integrated person region as the number of persons indicated in the integrated person region.

[0015] The person detection device according to the present application has an effect that each person can be detected even in a case where a plurality of persons overlap on an image. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a schematic configuration diagram of a vehicle control system in which the person detection device is installed (implemented).

[0017] Figure 2 is a hardware configuration diagram of an electronic control device that is one embodiment of the person detection device.

[0018] Figure 3 is a functional block diagram of a processor of the electronic control device relating to a vehicle control process including a person detection process.

[0019] Figure 4 Fig. 1 is an example of a graph indicating an integrated person region.

[0020] Figure 5 Fig. 2 is an example of a graph indicating a skeleton detection process and a mask process.

[0021] Figure 6 Fig. 3 is a workflow diagram of a person detection process.

[0022] Figure 7 Fig. 4 is a workflow diagram of a vehicle control process including the person detection process. DETAILED DESCRIPTION

[0023] Hereinafter, a person detection device and a person detection method executed on the person detection device and a person detection computer program will be described with reference to the drawings. The person detection device detects a region containing a person (hereinafter, sometimes referred to as a person region) on an image by inputting the image to a recognizer (discriminator) that is learned in advance to detect a person represented in the image. Also, the person detection device integrates two or more person regions overlapping by a predetermined number or more when the two or more person regions are detected, and integrates the person regions as one integrated person region. Further, the person detection device determines a skeleton of an undetermined and most front person among one or more persons contained in the integrated person region by inputting the integrated person region to a skeleton detector that is learned in advance to detect a skeleton of a person. Still further, the person detection device masks each of pixels representing the most front person in the integrated person region based on the skeleton of the most front person, and inputs the integrated person region after the masking to the skeleton detector again. Subsequently, the person detection device repeats the detection process of the skeleton and the process of the masking until a skeleton of a person is not detected from the integrated person region. Thereby, the person detection device can detect each person even when a plurality of persons overlap on the image.

[0024] Hereinafter, an example in which the person detection device is applied to a vehicle control system will be described. In the present example, the person detection device detects a detection target object existing around a vehicle by executing a person detection process on a series of images of a time series obtained by a camera mounted on the vehicle 10. The detection target object includes an object that has an influence on the travel of the vehicle 10, such as other vehicles, persons, road signs, signal machines, lane markings, and other objects on the road, which travel around the vehicle 10.

[0025] Figure 1 Fig. 1 is a schematic configuration diagram of a vehicle control system in which the person detection device is installed. In addition, Figure 2Fig. 1 is a hardware configuration diagram of an electronic control device that is an embodiment of a person detection device. In the present embodiment, a vehicle control system 1 mounted on a vehicle 10 and controlling the vehicle 10 has a camera 2 for taking an image of the surroundings of the vehicle 10 and an electronic control device (ECU) 3 that is an example of a person detection device. The camera 2 and the ECU 3 are communicably connected via an in-vehicle network that complies with a standard such as a controller area network. In addition, the vehicle control system 1 can also have a storage device that stores map information indicating the positions and kinds of map elements, lane marks, and the like for automatic driving control of the vehicle 10. Further, the vehicle control system 1 can also have a distance measuring sensor such as a LiDAR (Light Detection and Ranging) or a radar, a receiver that complies with a satellite positioning system and is used to measure the own position of the vehicle 10 such as a GPS receiver, and a wireless terminal for wireless communication with other devices. Still further, the vehicle control system 1 can also have a navigation device for searching for a travel scheduled route of the vehicle 10.

[0026] The camera 2 is an example of an imaging section and has a two-dimensional detector composed of an array of photoelectric conversion elements having sensitivity to visible light such as a CCD (Charge-Coupled Device) or a C-MOS (Complementary Metal-Oxide Semiconductor), and an imaging optical system that images an image of a region to be imaged on the two-dimensional detector. Also, the camera 2 is mounted in the vehicle 10, for example, in the vehicle cabin in a manner facing the front of the vehicle 10. Also, the camera 2 takes an image of a front region of the vehicle 10 at a predetermined imaging cycle (for example, 1 / 30 to 1 / 10 seconds) and generates an image representing the front region. The image obtained by the camera 2 is preferably a color image. In addition, the vehicle 10 can be provided with a plurality of cameras having different imaging directions or focal distances.

[0027] The camera 2 outputs the generated image to the ECU 3 via the in-vehicle network each time the image is generated.

[0028] The ECU 3 controls the vehicle 10. In the present embodiment, the ECU 3 controls the vehicle 10 to automatically drive the vehicle 10 on the basis of an object detected from a time-series series of images obtained by the camera 2. To this end, the ECU 3 has a communication interface (I / F) 21, a memory 22, and a processor 23.

[0029] The communication interface 21 is an example of a communication section and has an interface circuit for connecting the ECU 3 to the in-vehicle network. That is, the communication interface 21 is connected to the camera 2 via the in-vehicle network. Also, the communication interface 21 hands over the received image to the processor 23 each time the image is received from the camera 2.

[0030] The memory 22 is an example of a storage unit, and has, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. Also, the memory 22 stores various data and parameters used in the vehicle control processing including the person detection processing executed by the processor 23 of the ECU 3. Specifically, the memory 22 stores, for example, the image received from the camera 2, various parameters for determining the recognizer used in the person detection processing, and various parameters for determining the skeleton detector, the overlap degree threshold for the integration determination of the person region, and the like. Further, the memory 22 stores various data generated in the middle of the vehicle control processing for a certain period. Still further, the memory 22 can also store information such as map information used in the travel control of the vehicle 10.

[0031] The processor 23 is an example of a control unit, and has one or a plurality of CPUs (Central Processing Units) and peripheral circuits thereof. The processor 23 can also have other arithmetic circuits such as a logic operation unit, a numerical operation unit, or a GPU (Graphics Processing Unit). Also, the processor 23 executes the vehicle control processing including the person detection processing with respect to the image received from the camera 2 every time the image is received during the travel of the vehicle 10. Further, the processor 23 controls the vehicle 10 to automatically drive the vehicle 10 on the basis of the detected object around the vehicle 10.

[0032] Figure 3 is a functional block diagram of the processor 23 of the ECU 3 relating to the vehicle control processing including the person detection processing. The processor 23 has a detection unit 31, an overlap estimation unit 32, an integration unit 33, a skeleton detection unit 34, a mask unit 35, a repetition control unit 36, a drive planning unit 37, and a vehicle control unit 38. These respective units possessed by the processor 23 are, for example, functional modules realized by a computer program operating on the processor 23. Alternatively, these respective units possessed by the processor 23 can be dedicated arithmetic circuits provided to the processor 23. In addition, the detection unit 31, the overlap estimation unit 32, the integration unit 33, the skeleton detection unit 34, the mask unit 35, and the repetition control unit 36 among these respective units possessed by the processor 23 are included in the person detection processing. Further, in a case where a plurality of cameras are provided to the vehicle 10, the processor 23 can execute the person detection processing on the basis of the image obtained by each camera, for each camera.

[0033] The detection section 31 detects an object region containing a detection target object represented in the image every time an image is received from the camera 2 by inputting the latest received image to the recognizer for object detection, and determines the kind of the detection target object. In the present embodiment, a person is included as one of the detection target objects. In addition, the detection target objects can also include objects that can have an influence on the travel of the vehicle 10, such as other vehicles, traffic signals, road signs, and road markings. In addition, the object region containing a person is referred to as a person region as described above.

[0034] The detection section 31 uses, as the recognizer, a DNN (Deep Neural Network) that is previously learned to detect an object region containing a detection target object represented in an image, recognize the kind of the detection target object, and calculate a hidden confidence. The DNN used by the detection section 31 can be, for example, a DNN having a convolutional neural network (hereinafter, referred to as CNN) type architecture such as SSD (Single Shot MultiBox Detector) or Faster R-CNN. In this case, the recognizer is previously learned using a large number of teacher images representing any one of various kinds of detection target objects in accordance with a learning method such as the error back propagation method.

[0035] Alternatively, the detection section 31 can use, as the recognizer, a recognizer based on a machine learning method other than a neural network such as a support vector machine or AdaBoost. In this case, the recognizer is previously learned using a large number of teacher images as described above in accordance with a learning method corresponding to the applied machine learning method. In addition, in the case where such a recognizer is used, the detection section 31 sets windows having various positions, various sizes, or various aspect ratios in the image. Then, the detection section 31 calculates, for each window, a feature amount (for example, a Haar-like feature amount, a HOG feature amount, or the like) to be input to the recognizer from the window, and determines whether or not the detection target object is represented in the window by inputting the calculated feature amount to the recognizer. Further, the detection section 31 takes a window determined to represent any kind of detection target object as an object region. In particular, the detection section 31 takes a window determined to represent a person as a person region. Furthermore, the recognizer can be prepared for each kind of object that becomes a detection target.

[0036] The detection section 31 registers the position and the range of each detected object region on the image and the kind of the object contained in the object region in the detected object list. Further, the detection section 31 stores the detected object list in the storage 22.

[0037] The overlap estimation section 32 determines an overlap estimation region in which two or more persons are estimated to be present on the latest image by tracking persons detected in each of a series of past images obtained by the camera 2 in time series before the latest image.

[0038] For example, the overlap estimation section 32 tracks a person represented in each of a series of past images obtained by the camera 2 in time series by applying a tracking process based on optical flow such as the Lucas-Kanade method to each person region. Thus, the overlap estimation section 32 extracts a plurality of feature points from a person region by applying a filter for extracting feature points such as the SIFT (Scale-Invariant Feature Transform) or Harris operator to the person region in the most recent past image, for example. Also, the overlap estimation section 32 determines a corresponding point in the person region in a past image prior thereto in accordance with the tracking method applied, thereby calculating optical flow, for each of the plurality of feature points. Alternatively, the overlap estimation section 32 can track a person represented in each of a series of past images in time series by applying another tracking method suitable for tracking a moving object detected from an image to each region.

[0039] The overlap estimation section 32 predicts the position and range of a person region in which a person is represented in the latest image using the tracking results in a series of past images for each person being tracked. At this time, the overlap estimation section 32 predicts the position and range of the person region of the person in the latest image by performing a prediction process using a Kalman filter or a particle filter or the like on the person region of the person of interest in a series of past images. Alternatively, the overlap estimation section 32 can predict the position and range of the person region of the person in the latest image by extrapolating a locus of changes in the position and range of the person region of the person of interest in a series of past images.

[0040] The overlap estimation section 32 sets an overlap estimation region in a manner that includes two or more estimated person regions that overlap each other in a case where the predicted position and predicted range of the person region of two or more persons being tracked overlap at least partially in the latest image. For example, the overlap estimation section 32 sets, as the overlap estimation region, a circumscribed rectangular region of a region that is the union of the two or more estimated person regions that overlap each other or a region obtained by expanding the circumscribed rectangular region in the horizontal or vertical direction by a predetermined number of pixels. Also, the overlap estimation section 32 notifies the integration section 33 of the position and range of the overlap estimation region.

[0041] The integration unit 33 integrates two or more person regions into one integrated person region in a case where the two or more person regions overlap each other by a predetermined amount or more in the latest image obtained by the camera 2. In the present embodiment, the integration unit 33 determines whether to integrate two or more person regions into one integrated person region by performing non-maximum suppression (NMS) processing on the two or more person regions that overlap each other at least partially.

[0042] Specifically, the integration unit 33 calculates the degree of overlap between two or more person regions that overlap each other at least partially, and compares the calculated degree of overlap with a degree-of-overlap threshold value. The integration unit 33, for example, calculates the ratio (IoU: Intersection over Union) of the area of the overlapping region to the area of the set of two or more person regions that overlap each other at least partially as the degree of overlap. Alternatively, the integration unit 33 can calculate the ratio of the area of the overlapping region to the area of the largest person region among the two or more person regions that overlap each other at least partially as the degree of overlap. Further, the integration unit 33 determines to integrate the two or more person regions into one integrated person region in a case where the degree of overlap is a degree-of-overlap threshold value or more. On the other hand, the integration unit 33 determines that the two or more person regions respectively represent different persons and does not integrate the person regions in a case where the degree of overlap is less than the degree-of-overlap threshold value.

[0043] The integration unit 33 selects one person region (for example, a larger object region) from among the two or more person regions determined to be integrated into one integrated person region as the integrated person region.

[0044] Further, the integration unit 33 can set the degree-of-overlap threshold value applied in a case where two or more person regions that overlap each other at least partially are included in the overlap estimation region to a value lower than the degree-of-overlap threshold value applied in a case where the person regions are not included in the overlap estimation region. Thus, in the overlap estimation region estimated to represent two or more persons, it is easy to select a relatively large person region as the integrated person region. Therefore, the possibility that two or more persons are included in the integrated person region increases, so the integration unit 33 can suppress the failure of detection of any of the plurality of persons that appear to overlap on the image.

[0045] According to a modification, the integration section 33 can also take, as the integrated person region, a region that is a union of two or more person regions determined to be integrated into one integrated person region, or a circumscribed rectangular region of the region that is the union. Alternatively, the integration section 33 can take, as the integrated person region, a region that is a union of two or more person regions determined to be integrated into one integrated person region, or a circumscribed rectangular region of the region that is the union, in a case where the two or more person regions are contained in the overlap presumption region. On the other hand, in a case where the two or more person regions deviate from the overlap presumption region, the integration section 33 can also take one of the two or more person regions as the integrated person region.

[0046] Figure 4 is an example of an integrated person region. In Figure 4 In the image 400 shown on the left, two person regions 411 and 412 that partially overlap each other are detected. In this example, the degree of overlap of the person region 411 and the person region 412 is above the degree of overlap threshold, and therefore, as shown in the image 401 on the right, one integrated person region 421 that is a circumscribed rectangular region of a region that is a union of the person region 411 and the person region 412 is set.

[0047] The integration section 33 notifies the skeleton detection section 34 of the position and the range of the integrated person region.

[0048] The skeleton detection section 34 crops the integrated person region from the latest image obtained by the camera 2. Further, the skeleton detection section 34 detects the skeleton of the person that is not determined and is located at the most front side among one or more persons contained in the integrated person region, by inputting the cropped integrated person region to a skeleton detector that is learned in advance to detect the skeleton of a person. Further, the skeleton detection section 34 increases the number of skeletons detected from the integrated person region by one each time a skeleton is detected.

[0049] The skeleton detection section 34 can use a DNN having a CNN-type architecture as such a skeleton detector. The skeleton detector is learned in advance using a large number of teacher images that represent the skeleton of a person, in accordance with a learning method such as the error back propagation method. Further, the skeleton detector outputs the positions of a plurality of reference points for determining the skeleton such as the head, the neck, the shoulder, the elbow, the hand, the hip joint, the knee, and the foot, and a skeleton line that represents the connection relationship of the reference points in accordance with the body configuration. At this time, the skeleton detector calculates, for each pixel in the integrated person region, a confidence value that represents the certainty (accuracy) of displaying the reference point of each class (head, neck, etc.), by class, and outputs the position at which the confidence value is above a predetermined threshold as the reference point of the class. Further, the skeleton detector can detect the skeleton of the person that is not determined and is located at the most front side among the persons from the integrated person region, by selecting the reference point of each class from the detected reference points so that the certainty when connected in accordance with the body configuration is the maximum.

[0050] The skeleton detection section 34 notifies the repetition control section 36 of the result of the determination of whether or not a skeleton has been detected. Further, the skeleton detection section 34, in the case where a skeleton has been detected, notifies the mask section 35 of skeleton information indicating the positions of the respective reference points of the detected skeleton and the respective skeleton lines.

[0051] The mask section 35 updates the integrated person region by masking the region indicating the person at the most front side that has been newly detected, within the integrated person region, on the basis of the skeleton information notified from the skeleton detection section 34, i.e., the skeleton lines of the person at the most front side that has been newly detected. In the present embodiment, the mask section 35 takes the region within a predetermined distance from any of the skeleton lines as a mask region. Further, since the torso of a person is thicker than the hands and feet, the mask section 35 can also make the predetermined distance from the skeleton line corresponding to the torso larger than the predetermined distance from the skeleton line corresponding to the hands and feet. In addition, the smaller the integrated person region, the farther the person is presumed to be located from the camera 2, and therefore the mask section 35 can also make the predetermined distance from the skeleton line smaller as the integrated person region is smaller. Alternatively, the mask section 35 can also make the predetermined distance from the skeleton line smaller as the distance from the reference point corresponding to the head to the reference point corresponding to the feet is smaller.

[0052] The mask section 35 masks the mask region by replacing the values of the respective pixels included in the mask region with a certain value, or by replacing the values of the respective pixels included in the mask region so that the mask region becomes a predetermined pattern or random noise. Thereby, it becomes impossible to detect a reference point of a skeleton within the mask region. The mask section 35 hands over the integrated person region that has been updated by the mask region to the skeleton detection section 34.

[0053] The repetition control section 36 causes the skeleton detection section 34 and the mask section 35 to repeat the processing until a skeleton of a person is not detected within the integrated person region. In the present embodiment, the repetition control section 36, upon receiving the result of the determination from the skeleton detection section 34 indicating that a skeleton has been detected, causes the mask section 35 to update the integrated person region by masking the region indicating the detected person within the integrated person region. Further, the repetition control section 36 causes the skeleton detection section 34 to perform the skeleton detection processing again on the integrated person region that has been updated and output from the mask section 35. On the other hand, the repetition control section 36, upon receiving the result of the determination from the skeleton detection section 34 indicating that a skeleton has not been detected, takes the number of skeletons detected within the integrated person region by the counting by the skeleton detection section 34 at the point of time as the number of persons included in the integrated person region. Further, the repetition control section 36, by associating the person indicated within the integrated person region with the person related to the presumed position of the person region in the latest image among the persons being tracked, by the number of persons, can continue the tracking of the person.

[0054] Figure 5 is a diagram showing an example of the skeleton detection processing and the mask processing. By performing the first skeleton detection processing on the integrated person region 510 in the image 500 shown on the topmost side by the skeleton detection section 34, the skeleton 521 of the first person is detected as shown in the second image 501. Then, as shown in the third image 502, by performing the mask processing by the mask section 35, the integrated person region 511 in which the mask region 522 centered on the skeleton 521 is masked is obtained. By performing the second skeleton detection processing on this integrated person region 511, the skeleton 523 of the second person is detected from within the integrated person region 511 as shown in the fourth image 503. In this way, by alternately performing the skeleton detection processing and the mask processing, the skeletons of all the persons included in the integrated person region are sequentially detected.

[0055] Figure 6 is a workflow diagram of the person detection processing performed by the processor 23. The processor 23 performs the person detection processing in accordance with the workflow diagram shown in Figure 6 every time an image is received from the camera 2.

[0056] The detection section 31 of the processor 23 inputs an image obtained from the camera 2 to the recognizer and detects one or more persons represented in the image. That is, the detection section 31 detects one or more person regions including persons on the image (step S101).

[0057] The overlap estimation section 32 of the processor 23 sets an overlap estimation region on the latest image in such a manner that two or more predicted person regions at least partially overlap based on the person regions of each person being tracked in a series of past images in time series (step S102).

[0058] The integration section 33 of the processor 23 sets the overlap degree threshold value applied to the overlap estimation region to a value lower than the overlap degree threshold value applied to regions other than the overlap estimation region (step S103). Then, the integration section 33 integrates two or more person regions at least partially overlapping each other into one integrated person region, of which the overlap degree is the overlap degree threshold value or more, among the two or more person regions (step S104).

[0059] The skeleton detection section 34 of the processor 23 detects the skeleton of the person not detected and on the most front side by inputting the integrated person region to the skeleton detector (step S105).

[0060] The repetition control section 36 of the processor 23 determines whether a skeleton is detected in step S105 (step S106). In the case where a skeleton is detected (step S106: YES), the repetition control section 36 causes the mask section 35 of the processor 23 to mask the region indicating the detected person in the integrated person region on the basis of the detected skeleton (step S107). Further, the repetition control section 36 causes the processing of step S105 and the subsequent processing to be performed again on the integrated person region whose region indicating the detected person is masked.

[0061] On the other hand, in the case where a skeleton is not detected (step S106: NO), the repetition control section 36 takes the number of skeletons detected from the integrated person region at the point in time as the number of persons included in the integrated person region (step S108). The processor 23 then ends the person detection processing.

[0062] The drive planning section 37 refers to the detected object list, and generates one or more travel predetermined paths of the vehicle 10 in a predetermined section from the current position of the vehicle 10 to a predetermined distance (for example, 500 m to 1 km) ahead so that the objects around the vehicle 10 do not collide with the vehicle 10. The travel predetermined path is expressed as a set of target positions of the vehicle 10 at each time when the vehicle 10 travels in the predetermined section, for example.

[0063] The drive planning section 37 tracks the detected target objects (including persons. Hereinafter, simply referred to as objects) registered in the detected object list in order to generate the travel predetermined path, and predicts the trajectories of the tracked objects in the period from the current time to a predetermined time ahead.

[0064] For example, the drive planning section 37 tracks the object indicated in the object region by performing the same tracking processing as explained in the overlap estimation section 32 on the object region in the latest image obtained by the camera 2 and the object region in the past image.

[0065] The driving plan portion 37 converts the image inner coordinates of each object being tracked into coordinates on the bird's-eye image (bird's-eye coordinates) by performing viewpoint conversion processing using information about the mounting position of the camera 2 to the vehicle 10 and the like, for each object being tracked. At this time, the driving plan portion 37 can estimate the position of the detected object at the time each image was taken, based on the position and posture of the vehicle 10 at the time each image was taken, the estimated distance to the detected object, and the direction from the vehicle 10 toward the object. Further, the driving plan portion 37 can estimate the position and posture of the vehicle 10 at the time each image was taken, for example, based on current position information indicating the current position of the vehicle 10 obtained by a GPS receiver (not shown) mounted on the vehicle 10. Alternatively, the driving plan portion 37 can detect lane marks to the left and right of the vehicle 10 from each image obtained by the camera 2, and estimate the position and posture of the vehicle 10 by matching the detected lane marks with map information stored in the storage 22, each time an image is obtained by the camera 2. In addition, the driving plan portion 37 can determine the direction from the vehicle 10 toward the object based on the position of the object region on the image containing the detected object and the optical axis direction of the camera 2. Furthermore, it is assumed that the position of the lower end of the object region indicates the position at which the object represented in the object region meets the road surface. Thus, the driving plan portion 37 can estimate the distance to the object represented in the object region based on the orientation of the camera 2 and the set height of the camera 2 corresponding to the lower end of the object region. Moreover, the driving plan portion 37 can estimate the predicted trajectory of the object up to a predetermined time ahead by performing prediction processing using a Kalman Filter or a Particle Filter and the like on a series of bird's-eye coordinates for a predetermined period of time.

[0066] The driving plan portion 37 sets a travel predetermined path of the vehicle 10 based on the predicted trajectories of the objects being tracked, such that the predicted value of the distance between each of the objects being tracked and the vehicle 10 up to a predetermined time ahead is a predetermined distance or more for each object, and such that the travel predetermined path follows the travel predetermined route to the destination. At this time, the driving plan portion 37 calculates the reciprocal of the sum of the distances to the object being tracked closest to the position on the travel predetermined path at each time up to a predetermined time ahead as an evaluation function. Furthermore, the driving plan portion 37 can set the travel predetermined path such that the evaluation function is minimized according to a predetermined optimization method such as a dynamic programming method or a steepest descent method.

[0067] Further, the driving plan portion 37 can generate a plurality of travel predetermined paths. In this case, the driving plan portion 37 can select a path for which the sum of the absolute values of the accelerations of the vehicle 10 is the smallest, from among the plurality of travel predetermined paths.

[0068] The driving plan portion 37 notifies the vehicle control portion 38 of the generated travel predetermined path.

[0069] The vehicle control section 38 controls each section of the vehicle 10 to cause the vehicle 10 to travel along the notified travel predetermined path. For example, the vehicle control section 38 calculates a target acceleration of the vehicle 10 in accordance with the notified travel predetermined path and a current vehicle speed of the vehicle 10 measured by a vehicle speed sensor (not shown), and sets a throttle position or a brake amount so as to become the target acceleration. Also, the vehicle control section 38 calculates a fuel injection amount in accordance with the set throttle position, and outputs a control signal corresponding to the fuel injection amount to a fuel injection device of an engine of the vehicle 10. Alternatively, the vehicle control section 38 outputs a control signal corresponding to the set brake amount to a brake of the vehicle 10.

[0070] Further, the vehicle control section 38 calculates a steering angle of the vehicle 10 in accordance with the travel predetermined path in a case where the vehicle 10 changes a route of the vehicle 10 for traveling along the travel predetermined path. Also, the vehicle control section 38 outputs a control signal corresponding to the steering angle to an actuator (not shown) that controls a steering wheel of the vehicle 10.

[0071] Figure 7 is a workflow diagram of the vehicle control processing including the person detection processing executed by the processor 23. The processor 23 executes the vehicle control processing in accordance with the workflow diagram shown in Figure 7 each time an image is received from the camera 2.

[0072] The processor 23 detects one or more persons from an image obtained by the camera 2 in accordance with the workflow diagram shown in Figure 6 Further, the detection section 31 of the processor 23 detects a detection target object other than a person from the image (step S202).

[0073] The driving plan section 37 of the processor 23 estimates a predicted trajectory of each detection target object by tracking the object. Also, the driving plan section 37 generates a travel predetermined path of the vehicle 10 so as to be apart from the estimated predicted trajectory by a predetermined distance or more (step S203). The vehicle control section 38 of the processor 23 controls the vehicle 10 to cause the vehicle 10 to travel along the travel predetermined path (step S204). Then the processor 23 ends the vehicle control processing.

[0074] As described above, the person detection device integrates two or more person regions overlapping by or more than the predetermined degree detected from the image into one integrated person region. Also, the person detection device determines the skeleton of the person at the most front side among one or more persons included in the integrated person region by inputting the integrated person region to the skeleton detector previously learned to detect the skeleton of the person. Further, the person detection device masks the region indicating the person at the most front side within the integrated person region based on the skeleton of the person at the most front side, and inputs the integrated person region updated by the masking to the skeleton detector again. Subsequently, the person detection device repeats the skeleton detection processing and the masking processing until the skeleton of the person is not detected from the integrated person region. Thus, the person detection device can detect each person even in a case where a plurality of persons overlap on the image.

[0075] According to the modification, in a case where two or more person regions at least partially overlapping with each other and overlapping by or more than the overlap degree threshold deviate from the overlap presumption region, the integration section 33 can also determine that the same person is indicated within the two or more person regions. Also, the integration section 33 can select only one of the two or more person regions and delete the other person regions from the detection list. In this case, the processing of the skeleton detection section 34, the masking section 35, and the repetition control section 36 can also be omitted for the two or more person regions. Thus, the person detection device repeats the skeleton detection processing and the masking processing only for the region presumed to overlap a plurality of persons as a result of tracking from past images. Therefore, the person detection device can suppress the false detection of one person as a plurality of persons even when a plurality of person regions are detected for the same person. In this modification, the integration section 33 can also set the integrated person region in a manner to include the region becoming the union of the two or more person regions in a case where the two or more person regions overlapping by or more than the overlap degree threshold are included in the overlap presumption region, as described in the above-described embodiment. Thus, the person detection device can appropriately set the integrated person region depending on whether the two or more person regions overlapping by or more than the overlap degree threshold are included in the overlap presumption region. Therefore, the person detection device can not only suppress the false detection of one person as a plurality of persons, but also accurately detect each person of a plurality of persons appearing to overlap on the image.

[0076] The person detection device according to the above-described embodiment or modification can also be installed other than the in-vehicle device. For example, the person detection device according to the above-described embodiment or modification can also be configured to detect a person from an image generated by a monitoring camera disposed to capture a predetermined region outside or inside a house at a predetermined cycle. Also, in a case where a person is detected within a certain period, the person detection device can also cause a display connected to the person detection device to display a message indicating that a person is detected.

[0077] In addition, the computer program that realizes the functions of each part of the processor 23 of the person detection device according to the above-described embodiments or modified examples can also be provided in the form of a computer-readable removable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium.

[0078] As described above, various modifications can be made within the scope of the present application according to the embodiments implemented by those skilled in the art.

Claims

1. A human detection device, characterized by, having: a detection section that detects one or more person regions representing a person from an image generated by a photographing section; a merging section that, in a case where two or more of the person regions overlap by a degree that is equal to or greater than a predetermined threshold value, calculates, as the degree of overlap, a ratio of an area of an overlapping region to an area of a set of two or more of the person regions that at least partially overlap each other, by selecting one of the two or more person regions or setting a merged person region in a manner that includes the two or more person regions, or calculates, as the degree of overlap, a ratio of an area of an overlapping region to an area of a largest person region among the two or more person regions that at least partially overlap each other; a skeleton detection section that detects a skeleton of a person that is not determined and is the most front among one or more persons included in the merged person region, by inputting the merged person region to a skeleton detector that is learned in advance to detect a skeleton of a person, and increases a number of skeletons detected from within the merged person region by one each time a skeleton is detected; a mask section that, based on the skeleton of the most front person, updates the merged person region by masking a region representing the most front person within the merged person region, specifically, by setting a region within a predetermined distance from any one skeleton line as a mask region, masking the mask region by replacing values of each pixel included in the mask region with a certain value or by replacing values of each pixel included in the mask region so that the mask region becomes a predetermined pattern or random noise, and thereby becoming a reference point from which a skeleton is not detected from within the mask region, and thereby updating the merged person region by being masked by the mask region; and a repetition control section that, for the updated merged person region, repeatedly performs processing of the skeleton detection section and the mask section until a skeleton of a person is not detected, and sets the number as a number of persons represented in the merged person region.

2. The person detection device according to claim 1, further having an overlap estimation section that estimates an overlap estimation region representing two or more persons on the image by tracking persons detected from each of a series of past images in a time series generated by the photographing section before the image, the merging section determines that each of the two or more person regions represents the same person in a case where the two or more person regions that overlap by the degree that is equal to or greater than the threshold value deviate from the overlap estimation region.

3. The person detection device according to claim 2, wherein the merging section selects any one of the two or more person regions and deletes the other person region in a case where the two or more person regions that overlap by the degree that is equal to or greater than the threshold value deviate from the overlap estimation region, and on the other hand, sets the merged person region in a manner that includes a region that becomes a set of the two or more person regions in a case where the two or more person regions that overlap by the degree that is equal to or greater than the threshold value are included in the overlap estimation region. ​ 4. The person detection apparatus according to claim 1, wherein an overlap estimation section that estimates an overlap estimation region in which two or more persons are estimated to be present on the basis of tracking of persons detected from each of a series of past images in a time series generated by the imaging section before the image, and the integration section sets the threshold value lower in a case where the two or more person regions are included in the overlap estimation region than in a case where the two or more person regions are deviated from the overlap estimation region.

5. A method of human detection, characterized in that, comprises: detecting two or more person regions in which persons are present from an image generated by an imaging section; in a case where a degree of overlap of the two or more person regions is equal to or higher than a predetermined threshold value, calculating a ratio of an area of an overlapped region to an area of a set of the two or more person regions that at least partially overlap each other as the degree of overlap, or calculating a ratio of an area of an overlapped region to an area of a largest person region among the two or more person regions that at least partially overlap each other as the degree of overlap, by selecting one of the two or more person regions or setting an integrated person region in a manner of including the two or more person regions; detecting a skeleton of a most front person who is not determined among one or more persons included in the integrated person region by inputting the integrated person region to a skeleton detector that is learned in advance to detect a skeleton of a person, and increasing a number of skeletons detected from the integrated person region by one each time a skeleton is detected; updating the integrated person region by masking a region in which the most front person is present in the integrated person region on the basis of the skeleton of the most front person, specifically, by masking a mask region that is within a predetermined distance from any one of the skeleton lines by replacing values of pixels included in the mask region with a certain value or by replacing the values of the pixels included in the mask region so that the mask region becomes a predetermined pattern or random noise, whereby a reference point from which a skeleton is not detected in the mask region is made, and thereby the integrated person region is updated by being masked by the mask region; and repeating the detection of the skeleton and the masking of the region in which the person is present for the updated integrated person region until a skeleton of a person is not detected, and setting the number as a number of persons represented in the integrated person region.

6. A computer program product for person detection, comprising a computer program for person detection, characterized in that The person detection computer program is for causing a computer to execute: detecting two or more person regions in which persons are present from an image generated by an imaging section; in a case where a degree of overlap of the two or more person regions is equal to or higher than a predetermined threshold value, calculating a ratio of an area of an overlapped region to an area of a set of the two or more person regions that at least partially overlap each other as the degree of overlap, or calculating a ratio of an area of an overlapped region to an area of a largest person region among the two or more person regions that at least partially overlap each other as the degree of overlap, by selecting one of the two or more person regions or setting an integrated person region in a manner of including the two or more person regions; detecting a skeleton of an undetermined and most front person among one or more persons included in the integrated person region by inputting the integrated person region to a skeleton detector that is previously learned to detect a human skeleton, increasing a number of skeletons detected from the integrated person region by 1 each time a skeleton is detected; updating the integrated person region by masking a region indicating the most front person within the integrated person region based on the skeleton of the most front person, specifically, by setting a region within a predetermined distance from any one skeleton line as a mask region, masking the mask region by replacing values of each pixel included in the mask region with a certain value or by replacing values of each pixel included in the mask region so that the mask region becomes a predetermined pattern or random noise, whereby a reference point from which a skeleton is not detected within the mask region is made, and thereby the integrated person region is updated by being masked by the mask region, and repeating the detection of the skeleton and the masking of the region indicating the person until a skeleton of a person is not detected for the updated integrated person region, and setting the number as the number of persons indicated in the integrated person region. ​

Images