Object recognition system, computing device, automobile, vehicle lamp, and classifier learning method

Abstract

The object recognition system (10) includes a camera (30) and a processing device (40). The processing device (40) includes a classifier (42) that performs machine learning in a manner capable of recognizing an object based on an output image of the camera (30). The strobe camera (50) divides the depth direction into a plurality of ranges, and changes the time difference between light projection and exposure for each range to perform photography. The classifier (42) classifies a plurality of images (IMG 50 ) obtained by the strobe camera into a plurality of images (IMG 50 ) and a plurality of images (IMG 50 ) obtained by the strobe camera. 1 ~IMG N ) as learning data for machine learning.

Description

Technical Field

[0001] The present invention relates to an object recognition system. Background Art

[0002] In order to perform autonomous driving or automatic control of the light distribution of headlights, an object recognition system is used to sense the position and type of objects around the vehicle. The object recognition system includes a sensor and a processing device that analyzes the output of the sensor. Considering the purpose, required accuracy, and cost, the sensor is selected from cameras, LiDAR (Light Detection and Ranging: Light Detection and Ranging, Laser Imaging Detection and Ranging: Laser Imaging Detection and Ranging), millimeter wave radar, ultrasonic sonar, etc.

[0003] A general SLR camera cannot obtain depth information, so when multiple objects at different distances overlap, it is difficult to separate them.

[0004] A TOF camera is known as a camera that can obtain depth information. A TOF (Time Of Flight) camera projects infrared light through a light emitting device, measures the flight time until the reflected light returns to the image sensor, and obtains an image by converting the flight time into distance information.

[0005] [Prior art literature]

[0006] [Patent Document]

[0007] Patent Document 1: Japanese Patent Application Publication No. 2009-257983

[0008] Patent Document 2: International Publication WO2017 / 110413A1 Summary of the invention

[0009] [Problems to be solved by the invention]

[0010] (Item 1) The processing device includes a classifier. The classifier is optimized in advance by machine learning using learning data (also referred to as training data). The recognition rate of the classifier is greatly affected by the method of selecting the learning data.

[0011] The first aspect of the present invention is accomplished under the above circumstances, and one of the exemplary purposes of one solution is to improve the recognition rate of an object recognition system or reduce the learning cost.

[0012] (Items 2 and 3) The applicant proposes a sensor (hereinafter referred to as a gating camera in this specification) that replaces a TOF camera (Patent Documents 1 and 2). The gating camera divides the shooting range into multiple ranges, and performs multiple shots by changing the exposure timing and exposure time for each range. Thus, an image is obtained for each range of the object, and each image only includes the object included in the corresponding range.

[0013] The second aspect of the present invention is completed under the above situation, and one of the exemplary purposes of one solution is to provide an object recognition system using a strobe camera. In addition, the third aspect of the present invention is to improve the recognition rate of the object recognition system using a strobe camera or reduce the learning cost.

[0014] (Items 4 and 5) In an object recognition system using a strobe camera, since a plurality of images corresponding to a plurality of ranges need to be processed, the image processing load increases. Therefore, when a low-speed processing device is used, the frame rate decreases, and if the frame rate is to be increased, a high-speed, in other words, expensive, processing device is required.

[0015] The fourth and fifth aspects of the present invention are accomplished under the above circumstances, and one of the exemplary purposes of one scheme is to reduce the amount of computational processing and / or improve the recognition rate of an object recognition system using a gated camera.

[0016] [Technical solutions for solving technical problems]

[0017] 1. A scheme of the first aspect of the present invention relates to an object recognition system. The object recognition system includes: a camera; and a processing device including a classifier that performs machine learning in a manner capable of recognizing an object based on an output image of the camera. The classifier performs machine learning using a plurality of images obtained by a strobe camera as learning data, the strobe camera dividing the depth direction into a plurality of ranges and taking pictures by changing the time difference between light projection and exposure for each range.

[0018] 2. A second aspect of the present invention relates to an object recognition system. The object recognition system includes: a strobe camera that divides the depth direction into a plurality of ranges and changes the time difference between light projection and exposure for each range to perform photography; and a processing unit that can recognize the type of an object based on a plurality of images corresponding to the plurality of ranges obtained by the strobe camera. The processing unit includes a plurality of classifiers with different algorithms, and a classifier corresponding to a range is selected to perform object recognition.

[0019] 3. A third aspect of the present invention relates to an object recognition system. The object recognition system includes: a strobe camera that divides the depth direction into a plurality of ranges and changes the time difference between light projection and exposure for each range to perform photography; a scaler that scales a plurality of images corresponding to the plurality of ranges obtained by the strobe camera according to a coefficient specified for each range; and a classifier that can respectively identify the type of the object based on the plurality of scaled images.

[0020] 4. A scheme of the fourth aspect of the present invention relates to an object recognition system. The object recognition system includes: a strobe camera that divides the depth direction into a plurality of ranges, changes the time difference between light projection and exposure for each range to perform photography, and generates a plurality of images corresponding to the plurality of ranges; and an operation processing device that is configured to recognize the type of the object contained in each of the plurality of images under the condition that the position where the object may exist is limited to the interest region inherent in each of the plurality of images.

[0021] Another scheme of the fourth aspect of the present invention relates to a processing device. The processing device is used together with a strobe camera to form an object recognition system. The strobe camera divides the depth direction into multiple ranges, changes the time difference between light projection and exposure for each range to shoot, and generates multiple images corresponding to the multiple ranges. The processing device is configured to be able to identify the type of object contained in each of the multiple images under the restriction that the position where the object may exist is limited to the interest area inherent in each of the multiple images.

[0022] 5. A scheme of the fifth aspect of the present invention relates to an object recognition system. The object recognition system includes: a strobe camera that divides the depth direction into a plurality of ranges, changes the time difference between light projection and exposure for each range to perform photography, and generates a plurality of images corresponding to the plurality of ranges; and a processing device that is configured to define an allowable range of the size of an object image captured in the corresponding image for each range, and can recognize the type of the object image captured in each image based on the allowable range.

[0023] Another scheme of the fifth aspect of the present invention relates to a processing device. The processing device is used together with a strobe camera that divides the depth direction into multiple ranges, changes the time difference between light projection and exposure for each range to shoot, and generates multiple images corresponding to the multiple ranges, to form an object recognition system, and the processing device includes: a classifier that specifies the allowable range of the size of the object image captured in the corresponding image for each range, detects the type and size of the object image captured in each image, and generates multiple detection data corresponding to the multiple images; and a post-processing unit that receives the multiple detection data and determines whether the size of the object image contained in each detection data is included in the allowable range.

[0024] Another scheme of the fifth aspect of the present invention is also a processing device. The processing device is used together with a strobe camera that divides the depth direction into a plurality of ranges, changes the time difference between light projection and exposure for each range to shoot, and generates a plurality of images corresponding to the plurality of ranges, to form an object recognition system, and the processing device includes: a pre-processing unit that divides the plurality of images into sub-images of each object image shot by it; and a classifier that specifies an allowable range of the size of the object image shot in the corresponding image for each range, and determines the type of the object image contained in the sub-image when the size of the sub-image divided by the pre-processing unit is included in the allowable range.

[0025] [Effects of the invention]

[0026] According to an aspect of the present invention, it is possible to improve the recognition rate or reduce the learning cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a block diagram of the object recognition system according to the first embodiment.

[0028] Figure 2 A diagram for explaining the operation of the strobe camera.

[0029] Figure 3 (a) Figure 3 (b) is a diagram illustrating the acquisition of learning data using a strobe camera.

[0030] Figure 4 (a) is a diagram showing an image IMGx captured by a single-lens camera.

[0031] Figure 5 is a block diagram of a car including an object recognition system.

[0032] Figure 6 is a block diagram showing a vehicle lamp including an object detection system.

[0033] Figure 7 This is a block diagram of an object recognition system according to variation 1.

[0034] Figure 8 This is a block diagram of an object recognition system according to embodiment 2.

[0035] Fig. 9 A diagram for explaining the operation of the strobe camera.

[0036] Fig.10 (a) indicates Figure 8 The timing diagram of the object recognition system's actions, Fig.10 (b) is a timing diagram illustrating the operation of the object recognition system of the comparative technique.

[0037] Fig.11 is a block diagram of a car including an object recognition system.

[0038] Fig.12 is a block diagram showing a vehicle lamp including an object detection system.

[0039] Fig.13 This is a block diagram of an object recognition system according to a third embodiment.

[0040] Fig.14 A diagram for explaining the operation of the strobe camera.

[0041] Fig.15 (a) Fig.15 (b) is a diagram illustrating an image obtained by a gated camera.

[0042] Fig.16 (a) Fig.16 (b) is a diagram illustrating an image obtained by a gated camera.

[0043] Fig.17 (a)~ Fig.17 (c) is a diagram illustrating an example of scaling processing performed by a scaler.

[0044] Fig.18 is a block diagram of a car including an object recognition system.

[0045] Fig.19 is a block diagram showing a vehicle lamp including an object detection system.

[0046] Fig. 20 (a) Fig. 20 (b) is a diagram illustrating an image obtained by a gated camera.

[0047] Fig.21 This is a block diagram of an object recognition system of implementation mode 4-1.

[0048] Fig. 22 A diagram for explaining the operation of the strobe camera.

[0049] Fig.23 (a) Fig.23 (b) is a diagram illustrating an image obtained by a gated camera.

[0050] Fig.24 (a) Fig.24 (b) is a diagram illustrating ROI.

[0051] Fig.25 (a) Fig.25 (b) is an explanation Fig.21 Diagram of the object recognition system in action.

[0052] Fig.26 This is a block diagram of an object recognition system of implementation mode 4-2.

[0053] Fig. 27 (a) Fig. 27 (b) is an explanation Fig.26 Diagram of the object recognition system in action.

[0054] Fig.28 (a) Fig.28 (b) is a diagram illustrating a modification example of ROI.

[0055] Fig.29 (a) Fig.29 (b) is a diagram illustrating a modification example of ROI.

[0056] Fig.30 (a) Fig.30 (b) is a diagram illustrating the operation of the object recognition system of variant example 3.

[0057] Fig.31 is a block diagram of a car including an object recognition system.

[0058] Fig.32 is a block diagram showing a vehicle lamp including an object detection system.

[0059] Fig.33 This is a block diagram of an object recognition system of implementation mode 5-1.

[0060] Fig.34 A diagram for explaining the operation of the strobe camera.

[0061] Fig.35 (a) Fig.35 (b) is a diagram illustrating an image obtained by a gated camera.

[0062] Fig.36 (a) Fig.36 (b) is a diagram illustrating an image captured for each range.

[0063] Fig.37 This is a diagram showing the allowable range of dimensions.

[0064] Fig.38 (a) Fig.38 (b) is an explanation Fig.33 Diagram of the object recognition system in action.

[0065] Fig.39 This is a block diagram of an object recognition system of implementation mode 5-2.

[0066] Fig.40 (a) Fig.40 (b) is an explanation Fig.39 Diagram of the object recognition system in action.

[0067] Fig.41 is a block diagram of a car including an object recognition system.

[0068] Fig.42 is a block diagram showing a vehicle lamp including an object detection system. DETAILED DESCRIPTION

[0069] I Aspect 1 of the present invention

[0070] I-1. Summary

[0071] An embodiment associated with the first aspect of the present invention relates to an object recognition system. The object recognition system includes: a camera; and a processing device including a classifier that performs machine learning in a manner capable of recognizing an object based on an output image of the camera. The classifier performs machine learning using a plurality of images obtained by a strobe camera as learning data, the strobe camera dividing the depth direction into a plurality of ranges and photographing by changing the time difference between light projection and exposure for each range.

[0072] By using a strobe camera, multiple images of multiple objects overlapping in the depth direction can be obtained, and these images can be used as learning data to improve the recognition rate of each object. In addition, the so-called incremental effect can be obtained, which can reduce the learning cost.

[0073] The camera may also be a single-lens reflex camera. The camera may also be a strobe camera.

[0074] I-2. Detailed description

[0075] The first aspect of the present invention is described below based on a preferred embodiment and with reference to the accompanying drawings. In this specification, the same or equivalent components, members, and processes shown in the various drawings are marked with the same reference numerals, and repeated descriptions are appropriately omitted. In addition, the embodiments do not limit the invention but are exemplified, and not all features and combinations described in the embodiments are essential contents of the invention.

[0076] Figure 1 This is a block diagram of an object recognition system 10 according to Embodiment 1. This object recognition system 10 is mounted on a vehicle such as an automobile or a motorcycle, and determines the type (category) of an object OBJ existing around the vehicle.

[0077] The object recognition system 10 mainly includes a camera 30 and an operation processing device 40. The camera 30 is a monocular camera that takes pictures of the surroundings of the vehicle. The operation processing device 40 includes a classifier 42. The classifier 42 is implemented based on a learning completion model obtained through machine learning so that the object OBJ can be recognized based on the output image IMG of the camera 30. The algorithm of the classifier 42 is not particularly limited, and YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Region-based Convolutional Neural Network), SPPnet (Spatial Pyramid Pooling), Faster R-CNN, DSSD (Deconvolution–SSD), Mask R-CNN, etc. can be used, or an algorithm to be developed in the future can be used.

[0078] exist Figure 1 In addition to the object recognition system 10, the configuration for machine learning is also shown. A gate camera 50 is used in the machine learning of the classifier 42. The gate camera 50 is used to divide the depth direction into a plurality of N (N≧2) ranges RNG1 to RNG2. N The camera 50 generates a plurality of ranges RNG1 to RNG2. N Corresponding multiple images IMG1~IMG N By changing the type and direction of the object OBJ and photographing it with the strobe camera 50, a plurality of images IMG1 to IMG2 can be obtained for each type of the object OBJ. N A collection of .

[0079] The multiple images IMG1 to IMG N The set of images IMG1 to IMG2 is input to the computer 60 in correspondence with the type of the object OBJ. N The set of is used as learning data (training data) to perform machine learning of the classifier 42. The method of machine learning in the computer 60 can use a well-known technology, so the description is omitted. The learning result finally obtained by the computer 60 is provided to the operation processing device 40 to form the classifier 42.

[0080] The strobe camera 50 includes a projector 52, an image sensor 54, and a controller 56. The image captured by the strobe camera 50 divides the depth direction into a plurality of N (N≧2) ranges RNG1 to RNG NAdjacent ranges may overlap in the depth direction at their boundaries.

[0081] The light projector 52 irradiates the detection light L1 toward the front of the vehicle in synchronization with the light projection timing signal S1 supplied from the controller 56. The detection light L1 is preferably infrared light, but is not limited thereto and may be visible light having a predetermined wavelength.

[0082] The image sensor 54 is configured to generate an image IMG by performing exposure control in synchronization with a photographing timing signal S2 supplied from the controller 56. The image sensor 54 has sensitivity to the same wavelength as the detection light L1, and photographs reflected light (return light) L2 reflected from the object OBJ.

[0083] The controller 56 changes the light projection timing signal S1 and the image capture timing signal S2 for each range RNG, thereby changing the time difference between the light projection by the light projector 52 and the exposure of the image sensor 54. The camera 30 generates images corresponding to the plurality of ranges RNG1 to RNG2. N Corresponding multiple images IMG1~IMG N The i-th image IMG i Only the corresponding range RNG is captured i The objects contained in .

[0084] Figure 2 2 is a diagram for explaining the operation of the strobe camera 50. Figure 2 The RNG for the i-th range is shown in i The state when measuring. The projector 52 is synchronized with the light projection timing signal S1 and emits light during the light emission period τ1 between time t0 and t1. At the top, a graph of light is shown with the horizontal axis representing time and the vertical axis representing distance. i The distance of the nearest boundary is set to d MINi , will be in the range RNG i The distance from the deep side boundary is set to d MAXi .

[0085] At a certain moment, the light emitted from the projector 52 reaches a distance d MINi The round trip time T until the reflected light returns to the image sensor 54 is MINi T MINi =2×d MINi / c. c is the speed of light.

[0086] Similarly, at a certain moment, the light emitted from the projector 52 reaches the distance d MAXi The round trip time T until the reflected light returns to the image sensor 54 is MAXi T MAXi =2×d MAXi / c.

[0087] In the range RNG only i When the object OBJ contained in is photographed, the controller 56 takes the time t2=t0+T MINi The exposure starts at time t3 = t1 + T MAXi The shooting timing signal S2 is generated in such a way that the exposure is ended at the same time. This is one exposure operation.

[0088] In the i-th range RNG i When photographing, multiple exposures may be performed. In this case, the controller 56 may repeat the above-mentioned exposure operation multiple times at a predetermined cycle τ2.

[0089] The above is the configuration of the object recognition system 10. Next, the machine learning of the object recognition system 10 will be described.

[0090] Figure 3 (a) Figure 3 (b) is a diagram for explaining the acquisition of learning data using a strobe camera. In this example, N=2. When acquiring learning data, Figure 3 (a) Consider a situation where a plurality of objects OBJ1 and OBJ2 are arranged in different ranges RNG1 and RNG2. In this example, OBJ1 is a pedestrian and OBJ2 is a car.

[0091] exist Figure 3 In (b), it is shown that Figure 3 . The image IMG1 includes only the object OBJ1 included in the range RNG1 on the near side, and the image IMG2 includes only the object OBJ2 on the deep side. That is, when observed from the camera 30 of the object recognition system 10, the multiple objects OBJ1 and OBJ2 that appear to overlap can be separated. Furthermore, by inputting the multiple images IMG1 and IMG2 as different learning data into the computer 60, a classifier 42 that can recognize the objects OBJ1 and OBJ2 as different objects can be constructed.

[0092] The advantages of this embodiment will be made clear by comparison with the comparative technique. In the comparative technique, an image IMGx captured by a monocular camera is used as learning data, similarly to the camera 30 of the object recognition system 10 . Figure 4 (a) is a diagram showing an image IMGx captured by a single-lens camera. Figure 4 As shown in (a), in the image IMGx, two objects OBJ1 and OBJ2 overlap.

[0093] exist Figure 4In (b), a bounding box BB1 containing a pedestrian (OBJ1) is shown. Inside the bounding box BB1, there is not only information about the pedestrian (OBJ1) but also about the vehicle (OBJ2) in the background. Therefore, if Figure 4 If the image IMGx of (a) is used as learning data, learning will be advanced in a manner that includes information about the vehicle and can be recognized as a pedestrian OBJ1. In other words, it may be difficult to recognize a single pedestrian as a pedestrian without a vehicle in the background.

[0094] exist Figure 4 In (c), a bounding box BB2 including the vehicle (OBJ2) is shown. Inside the bounding box BB2, there is not only the vehicle (OBJ2) but also the information of the pedestrian (OBJ1) in the near front. Therefore, if Figure 4 If the image IMGx of (a) is used as learning data, learning is advanced so that the image OBJ2 can be recognized as a vehicle including information about pedestrians. In other words, it may be difficult to recognize a single vehicle without pedestrians in the front as a vehicle.

[0095] Therefore, according to the learning method of this embodiment, by using a strobe camera to separate the object OBJ1 on the near side and the object OBJ2 on the far side and learning them as different images, it is possible to learn the object OBJ1 in a manner that is capable of learning the object OBJ2. # (#=1、2) Each of them advances learning by recognizing only based on the information of its own OBJ1. i When it exists alone, or when it is combined with other objects OBJ j (i≠j) In the case of overlap, the object OBJ can be correctly identified i .

[0096] Machine learning requires a huge amount of learning data, and if the situation or conditions are changed to capture an image of an object, the learning cost increases. Therefore, research is usually conducted to increase one basic image to multiple images through "incrementation". Examples of general incremental techniques include left-right inversion, rotation, zooming in and out, brightness value changes, and contrast changes of the basic image. Figure 4 When the image (a) is used as the basic image, it is possible to increase the number of images to two by using a strobing camera. That is, the effect of increment can also be obtained by using a strobing camera.

[0097] Figure 5 1 is a block diagram of a car including an object recognition system 10. The car 300 includes headlights 302L and 302R. The object recognition system 10 is built into at least one of the headlights 302L and 302R. The headlight 302 is located at the front end of the car body and is most advantageous as a location for installing the camera 30 in addition to detecting surrounding objects.

[0098] Figure 6 2 is a block diagram showing a vehicle lamp 200 including an object detection system 210. The vehicle lamp 200 and the vehicle-side ECU 304 together constitute a lamp system 310. The vehicle lamp 200 includes a light source 202, a lighting circuit 204, and an optical system 206. Furthermore, the vehicle lamp 200 is provided with an object detection system 210. The object detection system 210 corresponds to the above-mentioned object recognition system 10, and includes a camera 30 and a processing unit 40.

[0099] The information related to the object OBJ detected by the calculation processing device 40 can also be used for the light distribution control of the vehicle lamp 200. Specifically, the lamp-side ECU 208 generates an appropriate light distribution pattern based on the information related to the type and position of the object OBJ generated by the calculation processing device 40. The lighting circuit 204 and the optical system 206 operate in a manner to obtain the light distribution pattern generated by the lamp-side ECU 208.

[0100] In addition, information on the object OBJ detected by the processing unit 40 may be transmitted to the vehicle-side ECU 304. The vehicle-side ECU may perform automatic driving based on the information.

[0101] Next, a modification example related to the first embodiment will be described.

[0102] (Variant 1)

[0103] Figure 7 1 is a block diagram of an object recognition system 10A according to Modification 1. The object recognition system 10A replaces Figure 1 The camera 30 includes a strobe camera 20. The strobe camera 20 is constructed as follows: Figure 1 The gate camera 20 is the same as the gate camera 50. The gate camera 20 is a camera that divides the depth direction into a plurality of M (M≧2) ranges RNG1 to RNG M The camera 20 generates a plurality of ranges RNG1 to RNG2. M Corresponding multiple images IMG1~IMG M .

[0104] The classifier 42 receives the information from the strobe camera 20 and the plurality of M (M≧2) ranges RNG1-RNG M Corresponding multiple images IMG1~IMG M The classifier 42 is configured to be able to classify the images based on the plurality of images IMG1 to IMG M The classifier 42 is trained using a plurality of images IMG1 to IMG2 obtained by the strobe camera 50. NThe number N of the ranges of the strobe camera 50 may be the same as or different from the number M of the ranges of the strobe camera 20 .

[0105] II. Second Aspect of the Invention

[0106] II-1. Summary

[0107] An embodiment associated with the second aspect of the present invention relates to an object recognition system for a vehicle. The object recognition system includes: a strobe camera that divides the depth direction into a plurality of ranges and changes the time difference between light projection and exposure for each range to perform photography; and a processing unit that can recognize the type of an object based on a plurality of images corresponding to the plurality of ranges obtained by the strobe camera. The processing unit includes a plurality of classifiers with different algorithms, and a classifier corresponding to a range is selected to perform object recognition.

[0108] According to this solution, the recognition rate can be improved by applying different algorithms to near objects and far objects.

[0109] The processing device can also use the classifier of the YOLO (You Only Look Once) algorithm in the image processing of the close range. YOLO is not suitable for the recognition of smaller objects, but has the advantage of high speed. Since the objects contained in the image are relatively large when the distance to the object is short, by using YOLO, objects contained in the close range can be detected / recognized with high accuracy and high speed.

[0110] The processing device can also use the classifier of the SSD (Single Shot MultiBox Detector) algorithm in the image processing of the distant range. Although the processing speed of SSD is poor, it has the advantage of being able to detect smaller objects with high accuracy. Since the objects contained in the image are relatively small in distance, by using SSD, objects contained in the distant range can be detected / identified with high accuracy.

[0111] In image processing in a relatively short range, the arithmetic processing device may use a classifier using a relatively high-speed algorithm.

[0112] Furthermore, in processing images in a relatively long range, the arithmetic processing device may use a classifier using a relatively high-precision algorithm.

[0113] II-2. Detailed description

[0114] Figure 8 This is a block diagram of an object recognition system 10 according to Embodiment 2. This object recognition system 10 is mounted on a vehicle such as an automobile or a motorcycle, and determines the type (category) of an object OBJ existing around the vehicle.

[0115] The object recognition system 10 mainly includes a strobe camera 20 and a processing device 40. The strobe camera 20 includes a projector 22, an image sensor 24, and a controller 26. The image captured by the strobe camera 20 divides the depth direction into a plurality of N (N≧2) ranges RNG1 to RNG2. N Adjacent ranges may overlap in the depth direction at their boundaries.

[0116] The light projector 22 irradiates the detection light L1 toward the front of the vehicle in synchronization with the light projection timing signal S1 supplied from the controller 26. The detection light L1 is preferably infrared light, but is not limited thereto and may be visible light having a predetermined wavelength.

[0117] The image sensor 24 is configured to generate an image IMG by performing exposure control in synchronization with a photographing timing signal S2 supplied from the controller 26. The image sensor 24 has sensitivity to the same wavelength as the detection light L1, and photographs reflected light (return light) L2 reflected from the object OBJ.

[0118] The controller 26 changes the light projection timing signal S1 and the photographing timing signal S2 for each range RNG, thereby changing the time difference between the light projection by the light projector 22 and the exposure of the image sensor 24. The strobe camera 20 generates images corresponding to the plurality of ranges RNG1 to RNG2. N Corresponding multiple images IMG1~IMG N The i-th image IMG i Only the corresponding range RNG is captured i The objects contained in .

[0119] Fig. 9 2 is a diagram for explaining the operation of the strobe camera 20. Fig. 9 The RNG for the i-th range is shown in i The state when measuring. The projector 22 is synchronized with the light projection timing signal S1 and emits light during the light emission period τ1 between time t0 and t1. At the top level, a graph of light is shown with the horizontal axis representing time and the vertical axis representing distance. i The distance of the nearest boundary is set to d MINi , will be in the range RNG i The distance from the deep side boundary is set to d MAXi .

[0120] At a certain moment, the light emitted from the projector 22 reaches a distance d MINi The round trip time T until the reflected light returns to the image sensor 24 is MINi T MINi =2×d MINi / c. c is the speed of light.

[0121] Similarly, at a certain moment, the light emitted from the projector 22 reaches a distance d MAXi The round trip time T until the reflected light returns to the image sensor 24 is MAXi T MAXi =2×d MAXi / c.

[0122] In the range RNG only i When the object OBJ contained in is photographed, the controller 26 takes the time t2=t0+T MINi The exposure starts at time t3 = t1 + T MAXi The shooting timing signal S2 is generated in such a way that the exposure is ended at the same time. This is one exposure operation.

[0123] In the i-th range RNG i When photographing, multiple exposures may be performed. In this case, the controller 26 may repeat the above-mentioned exposure operation multiple times at a predetermined cycle τ2.

[0124] return Figure 8 The calculation processing device 40 is configured to be able to obtain the information obtained by the strobe camera 20 and the plurality of ranges RNG1 to RNG N Corresponding multiple images IMG1~IMG N The processing device 40 can be implemented by a combination of a processor (hardware) such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), a microcomputer, and a software program executed by the processor (hardware). The processing device 40 can also be a combination of multiple processors. Alternatively, the processing device 40 can also be composed of hardware only.

[0125] The processing device 40 includes a plurality of M (M≧2) classifiers 42_1 to 42_M with different algorithms. The selector 44 supplies each image IMG to the classifier 42 selected according to the range RNG. The classifier 42 processes the supplied image IMG to detect and classify (recognize) the object contained therein.

[0126] The number M of classifiers 42 may be the same as the number N of range RNGs, or may be smaller than the number N.

[0127] One of the multiple algorithms may be a high-speed but low-precision algorithm (in other words, a small object detection accuracy is low). In addition, another of the multiple algorithms may be a low-speed but high-precision algorithm (in other words, a small object detection accuracy is high).

[0128] The processing device 40 can use a relatively high-speed and low-precision algorithm classifier for image processing in a relatively close range. Alternatively, the processing device 40 can use a relatively low-speed and high-precision algorithm classifier for image processing in a relatively far range.

[0129] For example, one of the multiple algorithms is preferably a YOLO (You Only Look Once) algorithm. In addition, another of the multiple algorithms is preferably an SSD (Single Shot MultiBox Detector) algorithm.

[0130] The above is the configuration of the object recognition system 10. Next, its operation will be described.

[0131] Fig.10 (a) indicates Figure 8 1 is a timing diagram of the operation of the object recognition system 10. Here, M=2, N=2, the classifier 42_1 of the YOLO algorithm is assigned to process the image IMG1 of the first range RNG1, and the classifier 42_2 of the SSD algorithm is assigned to process the image IMG2 of the second range RNG2.

[0132] The image IMG1 of the range RNG1 is acquired by the gate camera 20 and processed by the classifier 42_1 of the YOLO algorithm. Next, the image IMG2 of the range RNG2 is acquired by the gate camera 20 and processed by the classifier 42_2 of the SSD algorithm.

[0133] The above is the operation of the object recognition system 10. According to the object recognition system 10, by using classifiers of two or more algorithms that complement each other and assigning a classifier to each imaging range, it is possible to detect relatively close objects and relatively far objects with high accuracy in a short time.

[0134] In more detail, YOLO is not suitable for the recognition of smaller objects, but has the advantage of high speed. The objects contained in the image appear larger as the distance to the object is shorter. Therefore, by using YOLO in the image processing of the close range RNG1, objects contained in the close range can be detected / recognized with high accuracy and high speed.

[0135] In addition, SSD has a slower processing speed than YOLO, but has the advantage of being able to detect smaller objects with high accuracy. Objects contained in an image become smaller as the distance to the object increases, so by using SSD, objects contained in a distant range can be detected / identified with high accuracy.

[0136] Figure 8The advantages of the object recognition system 10 will be made clear by comparison with a comparative technique in which all objects are processed by a classifier of a high-precision algorithm (SSD) regardless of the range. Fig.10 (b) is a timing diagram illustrating the operation of the object recognition system of the comparative technology. If the SSD algorithm is applied to all images IMG1 and IMG2, high-precision detection can be performed regardless of the distance of the object. However, the detection time required to recognize objects in all ranges is much shorter than that of the image. Fig.10 In other words, it means that the frame rate is lowered in the comparative technique. If a high frame rate is to be achieved in the comparative technique, a high-speed CPU or microcomputer is required in the arithmetic processing device 40, and the cost of the object recognition system 10 increases.

[0137] Therefore, according to the object recognition system 10 of the second embodiment, by combining a plurality of algorithms, it is possible to achieve both high detection accuracy and a high frame rate without using such expensive (high-speed) hardware.

[0138] Fig.11 1 is a block diagram of a car including the object recognition system 10. The car 300 includes headlights 302L and 302R. The object recognition system 10 is built into at least one of the headlights 302L and 302R. The headlight 302 is located at the front end of the car body and is the most advantageous location for the strobe camera 20 in terms of detecting surrounding objects.

[0139] Fig.12 2 is a block diagram showing a vehicle lamp 200 including an object detection system 210. The vehicle lamp 200 and the vehicle-side ECU 304 together constitute a lamp system 310. The vehicle lamp 200 includes a light source 202, a lighting circuit 204, and an optical system 206. Furthermore, the vehicle lamp 200 is provided with an object detection system 210. The object detection system 210 corresponds to the above-mentioned object recognition system 10, and includes a strobe camera 20 and an operation processing device 40.

[0140] The information related to the object OBJ detected by the calculation processing device 40 can also be used for the light distribution control of the vehicle lamp 200. Specifically, the lamp-side ECU 208 generates an appropriate light distribution pattern based on the information related to the type and position of the object OBJ generated by the calculation processing device 40. The lighting circuit 204 and the optical system 206 operate in a manner to obtain the light distribution pattern generated by the lamp-side ECU 208.

[0141] In addition, information on the object OBJ detected by the processing unit 40 may be transmitted to the vehicle-side ECU 304. The vehicle-side ECU may perform automatic driving based on the information.

[0142] Next, a modification example related to the second embodiment will be described.

[0143] (Variant 1)

[0144] When M=2 and N≧3, the first to kth ranges RNG1 to RNG k Images IMG1~IMG k The processing is assigned to the classifier 42_1 of the YOLO algorithm, and the RNG in the k+1th to Nth range is k+1 ~RNG N Image IMG k+1 ~IMG N Processing, assign the classifier 42_2 of the SSD algorithm.

[0145] (Variant 2)

[0146] In Implementation 2, YOLO and SSD are described as multiple algorithms, but are not limited to these. As algorithms for object recognition, R-CNN (Region-based Convolutional Neural Network), SPPnet (Spatial Pyramid Pooling), Faster R-CNN, DSSD (Deconvolution–SSD), Mask R-CNN, etc. are known, and any one of them can be used, or, of course, an algorithm to be developed in the future can also be used.

[0147] III. Third Aspect of the Invention

[0148] III-1. Summary

[0149] An embodiment associated with the third aspect of the present invention relates to an object recognition system. The object recognition system includes: a strobe camera that divides the depth direction into a plurality of ranges and changes the time difference between light projection and exposure for each range to perform photography; a scaler that scales a plurality of images corresponding to the plurality of ranges obtained by the strobe camera according to a coefficient specified for each range; and a classifier that can respectively identify the type of the object based on the plurality of scaled images.

[0150] According to this solution, the sizes of the same objects can be made consistent in a plurality of images input to a classifier, thereby improving the recognition ability of the classifier or reducing the learning cost.

[0151] The coefficient can also be smaller in closer ranges and larger in farther ranges.

[0152] Alternatively, one of the multiple ranges may be set as the reference range. Alternatively, the coefficient of the reference range may be set to 1, the coefficient of the range closer than the reference range may be set to be less than 1, and the coefficient of the range farther than the reference range may be set to be greater than 1. Thus, the processing of the scaler can be simplified. In addition, when learning, the images captured by arranging objects within the reference range can be used as the focus, which can reduce the number of shots of objects arranged outside the reference range.

[0153] III-2. Detailed description

[0154] Hereinafter, the third aspect of the present invention will be described based on preferred embodiments with reference to the drawings.

[0155] Fig.13 This is a block diagram of an object recognition system 10 according to Embodiment 3. This object recognition system 10 is mounted on a vehicle such as an automobile or a motorcycle, and determines the type (category) of an object OBJ existing around the vehicle.

[0156] The object recognition system 10 mainly includes a strobe camera 20 and a processing device 40. The strobe camera 20 includes a projector 22, an image sensor 24, and a controller 26. The image captured by the strobe camera 20 divides the depth direction into a plurality of N (N≧2) ranges RNG1 to RNG2. N Adjacent ranges may overlap in the depth direction at their boundaries.

[0157] The light projector 22 irradiates the detection light L1 toward the front of the vehicle in synchronization with the light projection timing signal S1 supplied from the controller 26. The detection light L1 is preferably infrared light, but is not limited thereto and may be visible light having a predetermined wavelength.

[0158] The image sensor 24 is configured to generate an image IMG by performing exposure control in synchronization with a photographing timing signal S2 supplied from the controller 26. The image sensor 24 has sensitivity to the same wavelength as the detection light L1, and photographs reflected light (return light) L2 reflected from the object OBJ.

[0159] The controller 26 changes the light projection timing signal S1 and the photographing timing signal S2 for each range RNG, thereby changing the time difference between the light projection by the light projector 22 and the exposure of the image sensor 24. The strobe camera 20 generates images corresponding to the plurality of ranges RNG1 to RNG2. N Corresponding multiple images IMG1~IMG N The i-th image IMG i Only the corresponding range RNG is captured i The objects contained in .

[0160] Fig.14 2 is a diagram for explaining the operation of the strobe camera 20. Fig.14 The RNG for the i-th range is shown in i The state when measuring. The projector 22 is synchronized with the light projection timing signal S1 and emits light during the light emission period τ1 between time t0 and t1. At the top, a graph of light is shown with the horizontal axis representing time and the vertical axis representing distance. i The distance of the nearest boundary is set to d MINi , will be in the range RNG i The distance from the deep side boundary is set to d MAXi .

[0161] At a certain moment, the light emitted from the projector 22 reaches a distance d MINi The round trip time T until the reflected light returns to the image sensor 24 is MINi T MINi =2×d MINi / c. c is the speed of light.

[0162] Similarly, at a certain moment, the light emitted from the projector 22 reaches a distance d MAXi The round trip time T until the reflected light returns to the image sensor 24 is MAXi T MAXi =2×d MAXi / c.

[0163] In the range RNG only i When the object OBJ contained in is photographed, the controller 26 takes the time t2=t0+T MINi The exposure starts at time t3 = t1 + T MAXi The shooting timing signal S2 is generated in such a way that the exposure is ended at the same time. This is one exposure operation.

[0164] In the i-th range RNG i When photographing, multiple exposures may be performed. In this case, the controller 26 may repeat the above-mentioned exposure operation multiple times at a predetermined cycle τ2.

[0165] Fig.15 (a) Fig.15 (b) is a diagram illustrating an image obtained by the strobe camera 20. Fig.15 In the example of (a), there is an object (pedestrian) OBJ2 in range RNG2, and there is an object (vehicle) OBJ3 in range RNG3. Fig.15 In (b), it is shown that Fig.15 When the image IMG1 is captured, the image sensor is exposed only by the reflected light from the range RNG1 , so no object is captured in the image IMG1 .

[0166] When capturing image IMG2, since the image sensor is exposed only by the reflected light from range RNG2, only object OBJ2 is captured in image IMG2. Similarly, when capturing image IMG3, since the image sensor is exposed only by the reflected light from range RNG3, only object OBJ3 is captured in image IMG3. In this way, according to the strobe camera 20, it is possible to capture objects separately for each range.

[0167] return Fig.13 The calculation processing device 40 is configured to be able to obtain the information obtained by the strobe camera 20 and the plurality of ranges RNG1 to RNG N Corresponding multiple images IMG1~IMG N To identify the type of the object. The processing device 40 includes a classifier 42, which is implemented based on a model generated by machine learning. The algorithm of the classifier 42 is not particularly limited, and YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Region-based Convolutional Neural Network), SPPnet (Spatial Pyramid Pooling), Faster R-CNN, DSSD (Deconvolution–SSD), Mask R-CNN, etc. can be used, or an algorithm to be developed in the future can be used.

[0168] The operation processing device 40 can be implemented by a combination of a processor (hardware) such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), a microcomputer, and a software program executed by the processor (hardware). The operation processing device 40 can also be a combination of multiple processors. Alternatively, the operation processing device 40 can also be composed of hardware only.

[0169] The present inventors have studied the training of the classifier 42 and have found the following problems.

[0170] A large amount of learning data (training data) is used to train the classifier 42. The learning data uses image data captured by a strobe camera or a normal camera.

[0171] Fig.16 (a) Fig.16(b) is a diagram illustrating an image obtained by the strobe camera 20. Fig.16 As shown in (a) of FIG. 1 , consider a situation where the same object (here, a pedestrian) OBJ exists in different ranges. The strobe camera 20 has a horizontal angle of view (angle of view) θ and a vertical angle of view determined by its own sensor size and optical system. Since the strobe camera 20 takes pictures at the same angle of view regardless of the range, if the same object at different distances is photographed, the photographing magnification will be different depending on the distance to the object. Therefore, Fig.16 As shown in (b), the farther the object is, the smaller it is photographed, and the closer the object is, the larger it is photographed. In addition, this feature is not limited to the strobe camera, and is also the same when photographing with a normal camera.

[0172] The processing device 40 is required to have the ability to classify the objects OBJ1 to OBJ3 contained in any of the images IMG1 to IMG3 as pedestrians. Therefore, a method is adopted in which images of a certain category of objects are prepared at multiple different distances and used as learning data. In addition, a method (called increment) is also adopted in which a basic image captured at a certain distance is enlarged or reduced to generate multiple images and use them as learning data.

[0173] However, these methods may increase the learning cost. In addition, even if these methods are used, it is assumed that a sufficient recognition rate cannot be obtained.

[0174] Therefore, in this embodiment 3, if Fig.13 As shown, the operation processing device 40 includes a scaler 46 provided in the front stage of the classifier 42. The scaler 46 converts the image obtained by the strobe camera 20 into the image data of the plurality of ranges RNG1 to RNG N Corresponding multiple images IMG1~IMG N According to the RNG for each range # (#=1、2、…N) specified coefficient K # The relationship between the range and the coefficient can be maintained in a table or in the form of an expression.

[0175] Fig.17 (a)~ Fig.17 (c) is a diagram for explaining an example of the scaling process performed by the scaler 46. In this example, N=3. The coefficients K1 to K3 are the image IMG after the scaling process. S1 ~IMG S3 The size (the number of pixels x and y in the vertical and horizontal directions) of the object OBJ included in is determined so as to be close to the object OBJ.

[0176] The reference range may be selected from a plurality of ranges. And the coefficient of the reference range is determined to be 1 or near 1. # is the reference range, coefficient K # =1, it can make IMG # =IMGs # In this example, the central range RNG2 is the reference range, and the coefficient K2 is selected to be 1.

[0177] For a range where the distance from the strobe camera 20 is closer than the reference distance, K<1. # <1, the original image IMG # is reduced. When reducing, pixels may also be thinned out. The resulting image IMGs # The number of pixels is the same as the original image IMG # Compared with the previous period, Fig.17 In the example of (a), K1=0.8 is selected.

[0178] For a range where the distance from the strobe camera 20 is greater than the reference distance, K>1. # >1, the original image IMG # is enlarged. When enlarging, pixel interpolation is performed. The resulting image IMGs # The number of pixels is the same as the original image IMG # Compared with the increase. Fig.17 In the example of (c), K3=1.4 is selected.

[0179] return Fig.13 The image data IMGs1 to IMGs2 after the scaling process performed by the scaler 46 is input to the classifier 42. N The classifier 42 is based on the image data IMGs1 to IMGs N , respectively identify the type of object OBJ contained in each image data.

[0180] The above is the structure of the object recognition system 10. In the object recognition system 10, the image input to the classifier 42 is scaled so that the same object is close to the same size (number of pixels). Therefore, the classifier 42 can detect the object and recognize its category without relying on the distance to the object. This leads to an improvement in the recognition rate (accuracy) of the classifier 42.

[0181] By adopting the object recognition system 10, there is also an advantage of being able to reduce the learning cost of the classifier 42. That is, during learning, objects are mainly arranged in a range where the scaling coefficient is close to 1, and the images obtained at this time can be used as learning data for optimizing the parameters of the classifier 42. For a range where the scaling coefficient deviates greatly from 1, the number of shots can be reduced.

[0182] Fig.18 1 is a block diagram of a car including the object recognition system 10. The car 300 includes headlights 302L and 302R. The object recognition system 10 is built into at least one of the headlights 302L and 302R. The headlight 302 is located at the front end of the car body and is the most advantageous location for the strobe camera 20 in terms of detecting surrounding objects.

[0183] Fig.19 2 is a block diagram showing a vehicle lamp 200 including an object detection system 210. The vehicle lamp 200 and the vehicle-side ECU 304 together constitute a lamp system 310. The vehicle lamp 200 includes a light source 202, a lighting circuit 204, and an optical system 206. Furthermore, the vehicle lamp 200 is provided with an object detection system 210. The object detection system 210 corresponds to the above-mentioned object recognition system 10, and includes a strobe camera 20 and an operation processing device 40.

[0184] The information related to the object OBJ detected by the calculation processing device 40 can also be used for the light distribution control of the vehicle lamp 200. Specifically, the lamp-side ECU 208 generates an appropriate light distribution pattern based on the information related to the type and position of the object OBJ generated by the calculation processing device 40. The lighting circuit 204 and the optical system 206 operate in a manner to obtain the light distribution pattern generated by the lamp-side ECU 208.

[0185] In addition, information on the object OBJ detected by the processing unit 40 may be transmitted to the vehicle-side ECU 304. The vehicle-side ECU may perform automatic driving based on the information.

[0186] Next, a modification example related to the third embodiment will be described.

[0187] (Variant 1)

[0188] In the embodiment, the coefficient of the reference range is set to 1 based on the range near the center of the multiple ranges, and the coefficients of the ranges outside the reference range are smaller than 1 or larger than 1, but the present invention is not limited thereto. For example, the coefficient of the range on the deep side may be set to be larger than 1 based on the range near the front. On the contrary, for example, the coefficient of the range closer to the front side may be set to be smaller than 1 based on the range on the deepest side.

[0189] (Variant 2)

[0190] In the previous section, we set the reference range to one, but you can also set multiple reference ranges. Fig. 20 Modification example 2 is described. Fig. 20 (a) Fig. 20 (b) is a diagram illustrating an image obtained by the gated camera 20. Here, the number of ranges N is 4. The multiple ranges are divided into multiple sets. Each set includes at least one range, and a reference range is set for each set. For example, RNG1 and RNG2 are determined as the first set, and RNG3 and RNG4 are determined as the second set. When focusing on the first set, one of the ranges RNG1 and RNG2 included therein is determined as the reference range, and the coefficients of the remaining ranges are set so that the size of the object is close to the object size of the reference range.

[0191] Similarly, when focusing on the second set, one of the ranges RNG3 and RNG4 included therein is determined as the reference range, and the coefficients of the remaining ranges are set so that the size of the object approaches the size of the object in the reference range.

[0192] In the second modification, for each set of ranges, machine learning may be performed by emphasizing images obtained when the object is located in the reference range.

[0193] IV. Fourth Aspect of the Invention

[0194] IV-1. Overview

[0195] An embodiment associated with the fourth aspect of the present invention relates to an object recognition system. The object recognition system includes: a strobe camera that divides the depth direction into a plurality of ranges, changes the time difference between light projection and exposure for each range to perform shooting, and generates a plurality of images corresponding to the plurality of ranges; and an operation processing device that is configured to identify the type of object contained in each of the plurality of images under the restriction condition that an area where an object may exist is limited for each image.

[0196] The objects to be recognized are restricted in their locations according to their types. For example, traffic participants such as pedestrians and cars exist on the ground, not in the air. That is, the possible locations of traffic participants in the image are restricted to the area near the ground. In a strobe camera, when multiple images are taken at the same viewing angle, the area near the ground changes according to the distance from the camera, that is, for each image.

[0197] Therefore, in one embodiment, a unique region of interest is determined for each image, and the amount of computation can be reduced by limiting the scope of image processing of the classifier.

[0198] The object recognition system may also include: a pre-processing unit that cuts the images within the region of interest for each of the plurality of images; and a classifier that processes the output of the pre-processing unit. In this case, since the size of the image provided to the classifier is reduced, the amount of computational processing can be reduced.

[0199] The object recognition system may also include: a pre-processing unit that masks images outside the region of interest for each of the plurality of images; and a classifier that processes outputs of the pre-processing unit. The pre-processing unit may also paint the portion outside the region of interest with a single color.

[0200] In other embodiments, the position information of the object may be reflected in the determination of the type of the object at a later stage or within the classifier.

[0201] The computing processing device may also include: a classifier, which detects the types and positions of objects contained in multiple images respectively, and generates multiple detection data corresponding to the multiple images; and a post-processing unit, which receives multiple detection data and determines whether the position of the object contained in each detection data is included in the inherent interest area in the original image.

[0202] The region of interest may also be such that images of closer ranges are larger and images of farther ranges are smaller.

[0203] The height of the region of interest may be such that the image becomes smaller as the range becomes farther away. This can reduce the amount of computational processing for detecting objects near the ground.

[0204] The width of the region of interest may also be narrower as the image has a farther range.

[0205] IV-2. Detailed description

[0206] Hereinafter, a fourth aspect of the present invention will be described based on preferred embodiments with reference to the drawings.

[0207] (Implementation method 4-1)

[0208] Fig.21 This is a block diagram of an object recognition system 10 according to Embodiment 4-1. This object recognition system 10 is mounted on a vehicle such as an automobile or a motorcycle, and determines the type (category) of an object OBJ existing around the vehicle.

[0209] The object recognition system 10 mainly includes a strobe camera 20 and a processing device 40. The strobe camera 20 includes a projector 22, an image sensor 24, and a controller 26. The image captured by the strobe camera 20 divides the depth direction into a plurality of N (N≧2) ranges RNG1 to RNG2. N Adjacent ranges may overlap in the depth direction at their boundaries.

[0210] The light projector 22 irradiates the detection light L1 toward the front of the vehicle in synchronization with the light projection timing signal S1 supplied from the controller 26. The detection light L1 is preferably infrared light, but is not limited thereto and may be visible light having a predetermined wavelength.

[0211] The image sensor 24 is configured to generate an image IMG by performing exposure control in synchronization with a photographing timing signal S2 supplied from the controller 26. The image sensor 24 has sensitivity to the same wavelength as the detection light L1, and photographs reflected light (return light) L2 reflected from the object OBJ.

[0212] The controller 26 changes the light projection timing signal S1 and the photographing timing signal S2 for each range RNG, thereby changing the time difference between the light projection by the light projector 22 and the exposure of the image sensor 24. The strobe camera 20 generates images corresponding to the plurality of ranges RNG1 to RNG2. N Corresponding multiple images IMG1~IMG N The i-th image IMG i Only the corresponding range RNG is captured i The objects contained in .

[0213] Fig. 22 2 is a diagram for explaining the operation of the strobe camera 20. Fig. 22 The RNG for the i-th range is shown in i The state when measuring. The projector 22 is synchronized with the light projection timing signal S1 and emits light during the light emission period τ1 between time t0 and t1. At the top, a graph of light is shown with the horizontal axis representing time and the vertical axis representing distance. i The distance of the nearest boundary is set to d MINi , will be in the range RNG i The distance from the deep side boundary is set to d MAXi .

[0214] At a certain moment, the light emitted from the projector 22 reaches a distance d MINi The round trip time T until the reflected light returns to the image sensor 24 is MINi T MINi =2×d MINi / c. c is the speed of light.

[0215] Similarly, at a certain moment, the light emitted from the projector 22 reaches a distance d MAXi The round trip time T until the reflected light returns to the image sensor 24 is MAXi T MAXi =2×d MAXi / c.

[0216] In the range RNG onlyi When the object OBJ contained in is photographed, the controller 26 takes the time t2=t0+T MINi The exposure starts at time t3 = t1 + T MAXi The shooting timing signal S2 is generated in such a way that the exposure is ended at the same time. This is one exposure operation.

[0217] In the i-th range RNG i When photographing, multiple exposures may be performed. In this case, the controller 26 may repeat the above-mentioned exposure operation multiple times at a predetermined cycle τ2.

[0218] Fig.23 (a) Fig.23 (b) is a diagram illustrating an image obtained by the strobe camera 20. Fig.23 In the example of (a), there is an object (pedestrian) OBJ2 in range RNG2, and there is an object (vehicle) OBJ3 in range RNG3. Fig.23 In (b), it is shown that Fig.23 When the image IMG1 is captured, the image sensor is exposed only by the reflected light from the range RNG1 , so no object is captured in the image IMG1 .

[0219] When capturing image IMG2, since the image sensor is exposed only by the reflected light from range RNG2, only object OBJ2 is captured in image IMG2. Similarly, when capturing image IMG3, since the image sensor is exposed only by the reflected light from range RNG3, only object OBJ3 is captured in image IMG3. In this way, according to the strobe camera 20, objects can be separated for each range and captured.

[0220] return Fig.21 The calculation processing device 40 is configured to be able to obtain the information obtained by the strobe camera 20 and the plurality of ranges RNG1 to RNG N Corresponding multiple images IMG1~IMG NTo identify the type of the object. The processing device 40 includes a classifier 42, which is implemented based on a learning model generated by machine learning. The algorithm of the classifier 42 is not particularly limited, and YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Region-based Convolutional Neural Network), SPPnet (Spatial Pyramid Pooling), Faster R-CNN, DSSD (Deconvolution-SSD), Mask R-CNN, etc. can be used, or an algorithm to be developed in the future can be used.

[0221] The operation processing device 40 can be implemented by a combination of a processor (hardware) such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), a microcomputer, and a software program executed by the processor (hardware). The operation processing device 40 can also be a combination of multiple processors. Alternatively, the operation processing device 40 can also be composed of hardware only.

[0222] A unique region of interest (ROI) is determined for each image, that is, for each range. ROI is a region where an object may exist. The ROI corresponding to the i-th (i=1, 2, ... N) image is marked as ROI i For each image IMG i , the classifier 42 limits the possible location of the object OBJ to the image IMG i Region of Interest (ROI) i Within the constraints, identify the image IMG i The type of objects contained in .

[0223] For example, the output of the classifier 42 (referred to as detection data) OUT i (i=1, 2, ...N) contains the i-th image data IMG i The information on the location of each object and its type (category) is included in the . The type information may indicate the possibility (probability) of corresponding to a plurality of types, or may include an identifier of the type with the highest probability.

[0224] Fig.24 (a) Fig.24 (b) is a diagram illustrating ROI. Fig.24(a) is a view from the horizontal direction. Fig.24 (b) shows an image captured by the strobe camera 20. Fig.24 As shown in (a), the shooting range based on the strobe camera 20 is converted into a viewing angle, which is ±θ up and down with the horizontal axis as the center. When the height of the ground 2 is set to 0, in the i-th range RNG i Image IMG i In the photograph, the height H i Objects in the lower range.

[0225] On the other hand, the area where pedestrians or cars exist is limited to a specified range (0~H ROI ). In this way, it can be regarded as ROI ~H i There is no object to be detected in the range. Fig.24 As shown in (b), for each image IMG i , can be compared with H ROI The lower range is set as ROI i , will be higher than H ROI The upper range is set as the exclusion area (hatched line). Specifically, for the i-th image IMG i , from the bottom end, H ROI / H i The range of the ratio is set as ROI i The ROI is larger in the nearer range of the image, and smaller in the farther range of the image.

[0226] The height above the ground of the image sensor 24 of the strobe camera 20 is set to hc. i The distance from the deep side boundary is set to d MAXi In this case, H i =hc+d MAXi ×tanθ.

[0227] return Fig.21 In this embodiment, a pre-processing unit 46 is provided at the front stage of the classifier 42. The pre-processing unit 46 processes each image IMG i The classifier 42 sorts the cutout images IMGt1 to IMGt2 as outputs of the pre-processing unit 46. N to be processed.

[0228] The above is the configuration of the object recognition system 10. Next, its operation will be described. Fig.25 (a) Fig.25 (b) is an explanation Fig.21 FIG. 1 is a diagram illustrating the operation of the object recognition system 10 . Fig.25(a) represents the image IMG of the i-th range i , Fig.25 (b) represents the cropped image IMGt i .

[0229] exist Fig.25 (a) Image IMG i In the image, three graphics X, Y, and Z are captured. Graphic X is included in the ROI. i Graphics Y and Z are in ROI i Therefore, the probability that figure Y or Z is a pedestrian or a car is low.

[0230] like Fig.25 As shown in (b), in the image IMGt cut out by the pre-processing unit 46 i , only the graphic X is included. By processing the cut-out image by the classifier 42, the graphics Y and Z are excluded from the objects of the computational processing, and only the graphic X becomes the object of the recognition processing. Thus, the amount of computational processing of the computational processing device 40 can be reduced.

[0231] Furthermore, by excluding the graphics Y and Z from the objects to be processed, it is possible to prevent the graphics Y and Z from being mistakenly recognized as a car or a pedestrian. That is, the recognition rate of the object recognition system 10 can be improved.

[0232] (Implementation method 4-2)

[0233] In Embodiment 4-1, the object of operation processing is limited to the ROI by the pre-processing of the classifier 42, but it is not limited to this. In Embodiment 4-2, the position where the object OBJ may exist is limited to the region of interest inherent in each of the multiple images by the post-processing or internal processing of the classifier 42.

[0234] Fig.26 4-2 is a block diagram of an object recognition system 10A according to Embodiment 4-2. The processing device 40A includes a post-processing unit 48 provided after the classifier 42. The classifier 42 classifies the plurality of images IMG1 to IMG2 generated by the strobe camera 20. N , respectively generating detection data OUT1 to OUT2 representing the position and type of the object contained in each image N The post-processing unit 48 receives the detection data OUT1 to OUT2 generated by the classifier 42. N The post-processing unit 48 processes each detection data OUT i (i=1~N), determine whether the position of the object contained in it is included in the original image IMG i Inherent ROI i And, the position of the object is in the ROI i , remove the object.

[0235] Fig. 27 (a) Fig. 27 (b) is an explanation Fig.26 FIG. 1 is a diagram illustrating the operation of object recognition system 10A. Fig. 27 (a) is to convert the detection data OUT related to the i-th range i Schematically represented as a diagram of image data. Detection data OUT i Contains two bounding boxes X and Y. The probability that the object contained in one bounding box X is judged to be a pedestrian (Human) is 90%, and the probability that the object contained in the other bounding box Y is judged to be a car (car) is 80%.

[0236] As described above, the possibility that a car exists in the ROI is low. Therefore, it can be said that the probability that the object contained in the bounding box Y outside the ROI is a car is low. Therefore, the post-processing unit 48 removes the object outside the ROI and outputs the detection data OUT after the removal. i '.

[0237] According to Embodiment 4-2, the amount of computational processing of the classifier 42 cannot be reduced, but the recognition rate of the object can be improved.

[0238] Next, modifications related to Embodiments 4-1 and 4-2 are described.

[0239] (Variant 1)

[0240] Fig.28 (a) Fig.28 (b) is a diagram illustrating a modified example of ROI. Fig.28 As shown in (a) of FIG. 1 , each image sometimes includes a position below the ground. In this case, Fig.28 As shown in (b), except for the upper range of each image (H ROI ~H i ), the range below the ground 2 (<0) can also be treated as an area where no object is likely to exist. This can further reduce the amount of computational processing performed by the computational processing device 40.

[0241] (Variant 2)

[0242] In the description so far, the range where an object is unlikely to exist is excluded, but the present invention is not limited to this, and an area that does not need to be detected even if an object exists may be determined. Fig.29 (a) Fig.29 (b) is a diagram illustrating a modification example of ROI. Fig.29 (a) is the view from above. Fig.29 (b) shows an image captured by the strobe camera 20 .

[0243] In vehicle-mounted applications, the range of objects to be detected can also be limited in the left and right directions. For example, a pedestrian who is tens of meters away from the vehicle in the horizontal direction does not need to be detected immediately. Therefore, the detection range can also be determined in the horizontal direction with the vehicle as the center, and the detection range can be set for each image IMG. i , set the part corresponding to the detection range as ROI i This can further reduce the amount of computational processing.

[0244] Of course, it is also possible to combine shearing in the up-and-down direction with shearing in the left-and-right direction.

[0245] (Variant 3)

[0246] In the embodiment 4-1, the processing of the pre-processing unit 46 is not limited to the above-mentioned processing. The pre-processing unit 46 may also i The image outside the ROI is masked. For example, the pre-processing unit 46 may paint the outside of the ROI with a single color (such as black or white). The classifier 42 treats the masked image IMGm i as the processing object. Fig.30 (a) Fig.30 (b) is a diagram for explaining the operation of the object recognition system 10 according to the third modification.

[0247] exist Fig.30 (a) Image IMG i In the video, three graphics X, Y, and Z are captured. Graphic X is included in the ROI, while graphics Y and Z are outside the ROI. Therefore, it is less likely that graphics Y or Z is a pedestrian or a car.

[0248] like Fig.30 As shown in (b), the outside of the ROI is masked by the pre-processing unit 46, and the masked image IMGm i The images Y and Z are removed from the masked image IMGm by the classifier 42. i The processing is performed, and the graphics Y and Z are excluded from the objects of the calculation processing, and only the graphic X becomes the object of the recognition processing. In this way, the amount of calculation processing of the calculation processing device 40 can be reduced.

[0249] (Variant 4)

[0250] In the embodiment, the images of different ranges are processed by the same classifier 42, but the present invention is not limited thereto, and a different classifier may be used for each range.

[0251] (use)

[0252] Fig.311 is a block diagram of a car equipped with an object recognition system 10. The car 300 includes headlights 302L and 302R. The object recognition system 10 is built into at least one of the headlights 302L and 302R. The headlight 302 is located at the front end of the car body and is the most advantageous location for installing the strobe camera 20 in terms of detecting surrounding objects.

[0253] Fig.32 2 is a block diagram showing a vehicle lamp 200 having an object detection system 210. The vehicle lamp 200 and the vehicle-side ECU 304 together constitute a lamp system 310. The vehicle lamp 200 includes a light source 202, a lighting circuit 204, and an optical system 206. Furthermore, the vehicle lamp 200 is provided with an object detection system 210. The object detection system 210 corresponds to the above-mentioned object recognition system 10, and includes a strobe camera 20 and an operation processing device 40.

[0254] The information related to the object OBJ detected by the calculation processing device 40 can also be used for the light distribution control of the vehicle lamp 200. Specifically, the lamp-side ECU 208 generates an appropriate light distribution pattern based on the information related to the type and position of the object OBJ generated by the calculation processing device 40. The lighting circuit 204 and the optical system 206 operate in a manner to obtain the light distribution pattern generated by the lamp-side ECU 208.

[0255] In addition, information on the object OBJ detected by the processing unit 40 may be transmitted to the vehicle-side ECU 304. The vehicle-side ECU may perform automatic driving based on the information.

[0256] V. Fifth Aspect of the Invention

[0257] V-1. Overview

[0258] An embodiment associated with the fifth aspect of the present invention relates to an object recognition system. The object recognition system includes: a strobe camera that divides the depth direction into a plurality of ranges, changes the time difference between light projection and exposure for each range to perform photography, and generates a plurality of images corresponding to the plurality of ranges; and a processing device that specifies, for each range, an allowable range of the size of an object image captured in the corresponding image, and can recognize the type of the object image captured in each image based on the allowable range.

[0259] The size of the object image captured in the image changes according to the distance to the object. When a strobe camera is used as a shooting device, the distance to the object contained in one image can be obtained based on the information in the depth direction of the corresponding range. That is, when an object located in a certain range is captured, the size of the object image captured in the image is limited to a certain range. Therefore, by determining this range as the allowed range, the recognition rate of the object can be improved or the amount of calculation processing can be reduced.

[0260] The computing processing device may also include: a classifier, which detects the type and size of the object image captured by each of the multiple images, and generates multiple detection data corresponding to the multiple images; and a post-processing unit, which receives the multiple detection data and determines whether the size of the object image contained in each detection data is within an allowable range.

[0261] The computing processing device may also include: a pre-processing unit, which divides multiple images into sub-images for each object image captured by it; and a classifier, which determines the type of object image contained in the sub-image when the size of the sub-image divided by the pre-processing unit is within an allowable range.

[0262] The allowable range can also be specified in the vertical direction. The allowable range can also be specified in the horizontal direction.

[0263] The allowable range may also be individually specified for each type of object image, thereby further improving the recognition rate of the object.

[0264] The allowable range may also be larger for closer ranges and smaller for farther ranges. The height of the allowable range may also be smaller for farther ranges. The width of the allowable range may also be narrower for farther ranges.

[0265] V-2. Detailed description.

[0266] Hereinafter, the fifth aspect of the present invention will be described based on preferred embodiments with reference to the drawings.

[0267] (Implementation method 5-1)

[0268] Fig.33 This is a block diagram of an object recognition system 10C according to Embodiment 5-1. This object recognition system 10C is mounted on a vehicle such as an automobile or a motorcycle, and determines the type (category or also referred to as category) of an object OBJ existing around the vehicle.

[0269] The object recognition system 10C mainly includes a strobe camera 20 and a processing unit 40C. The strobe camera 20 includes a projector 22, an image sensor 24, and a controller 26. The depth direction is divided into a plurality of N (N≧2) ranges RNG1 to RNGN Adjacent ranges may overlap in the depth direction at their boundaries.

[0270] The light projector 22 irradiates the detection light L1 toward the front of the vehicle in synchronization with the light projection timing signal S1 supplied from the controller 26. The detection light L1 is preferably infrared light, but is not limited thereto and may be visible light having a predetermined wavelength.

[0271] The image sensor 24 is configured to generate an image IMG by performing exposure control in synchronization with a photographing timing signal S2 supplied from the controller 26. The image sensor 24 has sensitivity to the same wavelength as the detection light L1, and photographs reflected light (return light) L2 reflected from the object OBJ.

[0272] The controller 26 changes the light projection timing signal S1 and the photographing timing signal S2 for each range RNG, thereby changing the time difference between the light projection by the light projector 22 and the exposure of the image sensor 24. The strobe camera 20 generates images corresponding to the plurality of ranges RNG1 to RNG2. N Corresponding multiple images IMG1~IMG N The i-th image IMG i Only the corresponding range RNG is captured i The objects contained in .

[0273] Fig.34 2 is a diagram for explaining the operation of the strobe camera 20. Fig.34 The RNG for the i-th range is shown in i The state when measuring. The projector 22 is synchronized with the light projection timing signal S1 and emits light during the light emission period τ1 between time t0 and t1. At the top, a graph of light is shown with the horizontal axis representing time and the vertical axis representing distance. i The distance of the nearest boundary is set to d MINi , will be in the range RNG i The distance from the deep side boundary is set to d MAXi .

[0274] At a certain moment, the light emitted from the projector 22 reaches a distance d MINi The round trip time T until the reflected light returns to the image sensor 24 is MINi T MINi =2×d MINi / c. c is the speed of light.

[0275] Similarly, at a certain moment, the light emitted from the projector 22 reaches a distance d MAXi The round trip time T until the reflected light returns to the image sensor 24 is MAXi T MAXi =2×d MAXi / c.

[0276] In the range RNG only i When the object OBJ contained in is photographed, the controller 26 takes the time t2=t0+T MINi The exposure starts at time t3 = t1 + T MAXi The shooting timing signal S2 is generated in such a way that the exposure is ended at the same time. This is one exposure operation.

[0277] In the i-th range RNG i When photographing, multiple exposures may be performed. In this case, the controller 26 may repeat the above-mentioned exposure operation multiple times at a predetermined cycle τ2.

[0278] Fig.35 (a) Fig.35 (b) is a diagram illustrating an image obtained by the strobe camera 20. Fig.35 In the example of (a), there is an object (pedestrian) OBJ2 in range RNG2, and there is an object (vehicle) OBJ3 in range RNG3. Fig.35 In (b), it is shown that Fig.35 When the image IMG1 is captured, the image sensor is exposed only by the reflected light from the range RNG1 , so no object image is captured in the image IMG1 .

[0279] When capturing image IMG2, since the image sensor is exposed only by the reflected light from range RNG2, only object OBJ2 is captured in image IMG2. Similarly, when capturing image IMG3, since the image sensor is exposed only by the reflected light from range RNG3, only object OBJ3 is captured in image IMG3. In this way, according to the strobe camera 20, it is possible to capture objects separately for each range.

[0280] return Fig.33 The calculation processing device 40C is configured to be able to obtain the information obtained by the strobe camera 20 and the plurality of ranges RNG1 to RNG N The corresponding multiple images IMG1~IMG NTo identify the type of the object. The processing device 40C includes a classifier 42, which is implemented based on a learning completion model generated by machine learning. The algorithm of the classifier 42 is not particularly limited, and YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Region-based Convolutional Neural Network), SPPnet (Spatial Pyramid Pooling), Faster R-CNN, DSSD (Deconvolution-SSD), Mask R-CNN, etc. can be used, or an algorithm to be developed in the future can be used.

[0281] The operation processing device 40C can be implemented by a combination of a processor (hardware) such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), a microcomputer, and a software program executed by the processor (hardware). The operation processing device 40C may also be a combination of multiple processors. Alternatively, the operation processing device 40C may also be composed of hardware only.

[0282] The classifier 42 identifies the type of object image included in each of the plurality of images IMG under the premise (constraint) that the size of the object image to be detected is within the allowable range.

[0283] For example, the output of the classifier 42 (referred to as detection data) OUT i (i=1, 2, ...N) contains the i-th image data IMG i The size information of each object image and its type (category) information included in the detection data OUT. When the detection data OUT includes the information of the bounding box of each object, the height and width of the bounding box can be used as the size information of the object. The type information can indicate the possibility of corresponding to multiple types (belonging probability), or it can include the identifier of the type with the highest belonging probability.

[0284] Fig.36 (a) Fig.36 (b) is a diagram illustrating an image captured for each range. Fig.36 (a) is a view from the horizontal direction. Fig.36 (b) shows an image captured by the strobe camera 20. Fig.36As shown in (a), the shooting range of the strobe camera 20 is converted into a viewing angle, which is ±θ up and down with the horizontal axis as the center. Fig.36 As shown in (b), the size of the object image captured in each image becomes smaller as the range becomes farther.

[0285] Fig.37 is a diagram illustrating the allowable range. Here, the allowable range is defined in the vertical direction. Focusing on the i-th range RNG i The object image is captured at its largest when the object OBJ is located at the boundary on the near side (A), and the object image is captured at its smallest when the object OBJ is located at the boundary on the far side (B).

[0286] Set the minimum height of object OBJ to h MIN , set the maximum value to h MAX Consider that the object OBJ exists at position (A) and its maximum height is h MAX When you put the image IMG i When the number of pixels in the vertical direction is set to Y, the number of pixels y in the vertical direction of the object in the image is

[0287] y MAXi =Y×h MAX / (d MINi ×tanθ×2)

[0288] The MAXi Give range RNG i The maximum value within the allowed range.

[0289] Consider that object OBJ exists at position (B) and its maximum height is h MIN At this time, the number of pixels y in the vertical direction of the object in the image MINi for

[0290] y MINi =Y×h MIN / (d MAXi ×tanθ×2)

[0291] The MINi Give range RNG i The minimum value within the allowed range.

[0292] That is, the allowed range (number of pixels in the vertical direction) is y MINi ~y MAXi In addition, the method for determining the allowable range is not limited to the above method.

[0293] return Fig.33The processing device 40C may exclude the object image from the processing target when the size of the detected object image deviates from the allowable range. Alternatively, the relationship between the size of the detected object image and the allowable range may be reflected in the probability of belonging to the type (category / section).

[0294] In the present embodiment, a post-processing unit 48 is provided at the subsequent stage of the classifier 42. The classifier 42 processes the plurality of images IMG1 to IMG N , respectively detect the type and size of the object image contained therein, and generate a plurality of detection data OUT1 to OUT N .

[0295] The post-processing unit 48 receives the detection data OUT1 to OUT N The post-processing unit 48 processes each detection data OUT i (i=1~N), determine whether the size of the object image contained in it is included in its range RNG i And, based on the determination result, the detection data OUT is corrected. i .

[0296] For example, the post-processing unit 48 may also detect the size of an object image outside the allowable range from the detection data OUT. i Alternatively, the post-processing unit 48 may reduce the probability of detecting that the object belongs to a certain type (category, category) when the size of a certain object image deviates from the allowable range.

[0297] The above is the configuration of the object recognition system 10C. Next, its operation will be described. Fig.38 (a) Fig.38 (b) is an explanation Fig.33 FIG. 1 is a diagram illustrating the operation of the object recognition system 10C. Fig.38 (a) will be the same as the i-th range RNG i Related test data OUT i Schematically represented as image data, Fig.38 (b) The corrected detection data OUT i ' is schematically represented as image data.

[0298] like Fig.38 As shown in (a), the detection data OUT before correction iIt includes three bounding boxes α, β, and γ. The probability that the object included in bounding box α is determined to belong to a pedestrian (Human) is 90%. The probability that the object included in bounding box β is determined to belong to a pedestrian (Human) is 60%. Bounding box γ contains an image of an object with a poster depicting a person pasted on a utility pole or a building. The probability that this object is determined to belong to a pedestrian (Human) is 20%.

[0299] For the object included in bounding box α, its height h satisfies y MINi <h X <y MAXi The relationship. Therefore, it can be said that the accuracy of the object within bounding box α as the detection target (Human) is high. Therefore, the corrected detection data OUT i includes the object of bounding box α.

[0300] For the object included in bounding box β, its height h is h < y MINi , deviating from the allowable range. Therefore, it can be said that the probability of this object being the detection target is low. For the object included in bounding box γ, its height h is y MAXi <h, deviating from the allowable range. Therefore, according to the corrected detection data OUT i ’, it can be said that the probability of the objects included in bounding boxes β and γ being the detection target is low. Therefore, bounding boxes β and γ can be removed from the corrected detection data OUT i ’.

[0301] The above is the operation of the object recognition system 10C. According to the object recognition system 10C, the recognition rate of objects can be improved.

[0302] (Embodiment 5-2)

[0303] Fig.39 is a block diagram of the object recognition system 10D of Embodiment 5-2. The arithmetic processing unit 40D includes a classifier 42 and a preprocessing unit 46 provided at its front stage.

[0304] The preprocessing unit 46 divides multiple images IMG1 to IMG N into sub-images of each object image included therein respectively.

[0305] Similar to Embodiment 5-1, for multiple ranges RNG1 to RNG N the allowable range of the size of the object image captured in the corresponding image is specified respectively.

[0306] The classifier 42 is configured to be able to determine the type of object image included in each sub-image SUB. When the size of the sub-image SUB divided by the pre-processing unit 46 is within the allowable range, the classifier 42 determines the type of object image included in the sub-image SUB.

[0307] Fig.40 (a) Fig.40 (b) is an explanation Fig.39 FIG. 1 is a diagram illustrating the operation of the object recognition system 10D. Fig.40 (a) represents the RNG with the i-th range i The image IMG obtained by association i , Fig.40 (b) represents the image IMG i Multiple sub-images SUB i-1 ~SUB i-3 In this example, the original image IMG i Contains 3 objects.

[0308] About sub-image SUB i-1 , whose height (number of pixels) exceeds the maximum y value allowed MAXi On the contrary, regarding the sub-image SUB i-3 , whose height (number of pixels) is less than the minimum value y of the allowed range MINi Therefore, the classifier 42 removes the sub-image SUB before performing the object recognition operation. i-1 、Sub-image SUB i-3 About sub-image SUB i-2 , its height (number of pixels) is within the allowable range. Therefore, it becomes the object of recognition operation performed by the classifier 42.

[0309] The above is the operation of the object recognition system 10D. According to the object recognition system 10D, the amount of computational processing of the computation processing device 40D can be reduced by filtering the sub-images to be the target of the recognition computation by the classifier 42 based on the size.

[0310] In addition, by removing the sub-image SUB from the object to be processed i-1 、Sub-image SUB i-3 , which can prevent the sub-image SUB i-1 、Sub-image SUB i-3 That is, the recognition rate of the object recognition system 10D can be improved.

[0311] The present invention has been described above based on the embodiments. It should be understood by those skilled in the art that the embodiments are illustrative, and the combination of its constituent elements or processing flows may have various variations, and such variations are also included in the scope of the present invention. Such variations are described below.

[0312] (Variant 1)

[0313] The allowed range can also be considered in the horizontal direction.

[0314] Based on the shooting range of the strobe camera 20, the angle of view is converted to In addition, the minimum horizontal width of the object is set to w MIN , the maximum value is set to w MAX At this time, a certain range of RNG i The minimum value of the allowed range X MINi , maximum value X MAXi It can be expressed as follows.

[0315]

[0316]

[0317] The calculation processing device may determine whether the size of the detected object image is within the allowable range only in the vertical direction, only in the horizontal direction, or in both directions.

[0318] (Variant 2)

[0319] In the embodiment, the images of different ranges are processed by the same classifier 42, but the present invention is not limited thereto, and a different classifier may be used for each range.

[0320] (Variant 3)

[0321] The permissible range may also be set individually for each type of object. For example, when pedestrians and cars are detected, different permissible ranges may be set for the width of pedestrians and the width of cars.

[0322] (Variant 4)

[0323] In Embodiments 5-1 and 5-2, the upper limit and the lower limit are defined for the allowable range, but only one of them may be defined.

[0324] (use)

[0325] Fig.411 is a block diagram of a car including the object recognition system 10. The car 300 includes headlights 302L and 302R. The object recognition system 10 is built into at least one of the headlights 302L and 302R. The headlight 302 is located at the front end of the car body and is the most advantageous location for the strobe camera 20 in terms of detecting surrounding objects.

[0326] Fig.42 2 is a block diagram showing a vehicle lamp 200 including an object detection system 210. The vehicle lamp 200 and the vehicle-side ECU 304 together constitute a lamp system 310. The vehicle lamp 200 includes a light source 202, a lighting circuit 204, and an optical system 206. Furthermore, the vehicle lamp 200 is provided with an object detection system 210. The object detection system 210 corresponds to the above-mentioned object recognition system 10, and includes a strobe camera 20 and an operation processing device 40.

[0327] The information related to the object OBJ detected by the calculation processing device 40 can also be used for the light distribution control of the vehicle lamp 200. Specifically, the lamp-side ECU 208 generates an appropriate light distribution pattern based on the information related to the type and position of the object OBJ generated by the calculation processing device 40. The lighting circuit 204 and the optical system 206 operate in a manner to obtain the light distribution pattern generated by the lamp-side ECU 208.

[0328] In addition, information on the object OBJ detected by the processing unit 40 may be transmitted to the vehicle-side ECU 304. The vehicle-side ECU may perform automatic driving based on the information.

[0329] The present invention has been described using specific words based on the implementation method, but the implementation method only represents one aspect of the principle and application of the present invention. In the implementation method, many modifications and configuration changes can be confirmed without departing from the scope of the idea of ​​the present invention defined in the claims.

[0330] [Industrial Applicability]

[0331] The present invention relates to an object recognition system.

[0332] [Explanation of Reference Numerals]

[0333] S1 light projection timing signal, S2 photography timing signal, 10 object recognition system, 20 gate camera, 30 camera, 40 operation processing device, 42 classifier, 50 gate camera, 52 light projector, 54 image sensor, 56 controller, 60 computer, 200 vehicle lamp, 202 light source, 204 lighting circuit, 206 optical system, 300 automatic vehicle, 302 headlight, 304 vehicle-side ECU, 310 lighting system.

Claims

1. An object recognition system, characterized in that: include: Camera, and an operation processing device including a classifier for performing machine learning in a manner capable of recognizing an object based on an output image of the camera; The classifier uses a plurality of images obtained by a strobe camera and the types of objects as learning data for machine learning. The strobe camera divides the depth direction into a plurality of ranges and takes pictures with different time differences between projection and exposure for each range. The processing device includes a plurality of classifiers with different algorithms, and a classifier corresponding to the range is selected to perform object recognition. The processing device uses a classifier with a relatively high-speed algorithm in image processing in a relatively close range. The processing device is configured to identify the type of the object contained in each of the plurality of images under the condition that the location of the object is limited to a region of interest inherent in each of the plurality of images. The processing device further includes a pre-processing unit, which cuts the images in the region of interest respectively for the plurality of images. The classifier processes the output of the pre-processing unit, and detects the types and positions of the objects contained in the multiple images respectively, and generates multiple detection data corresponding to the multiple images. The arithmetic processing device further includes a post-processing unit that receives the plurality of detection data and determines whether the position of the object included in each detection data is included in the interest region inherent in the original image.

2. The object recognition system according to claim 1, characterized in that: The camera is a single-lens reflex camera.

3. The object recognition system according to claim 1, wherein: The camera is a strobed camera.

4. The object recognition system according to claim 1, wherein: The arithmetic processing device uses a classifier of the YOLO algorithm in the image processing of a close range.

5. The object recognition system according to claim 1, characterized in that: The arithmetic processing device uses a classifier of an SSD algorithm in long-range image processing.

6. The object recognition system according to claim 1, wherein: The arithmetic processing device uses a classifier of a relatively high-precision algorithm in image processing at a relatively long range.

7. A car, characterized in that: include: An object recognition system as claimed in any one of claims 1 to 6.

8. A vehicle lamp, characterized in that: include: An object recognition system as claimed in any one of claims 1 to 6.

9. A method for learning multiple classifiers with different algorithms for identifying objects based on an output image of a camera, characterized in that: include: The steps of dividing the depth direction into a plurality of ranges using a gated camera, and photographing each range with different light projection and exposure time differences, and The step of using the plurality of images obtained by the strobe camera in correspondence with the types of objects as learning data, and causing the classifier corresponding to the range among the plurality of classifications to perform machine learning; in A classifier that uses a relatively high-speed algorithm to process images in a relatively close range. Under the condition that the location of the object is limited to the region of interest inherent in each of the plurality of images, the type of the object contained in each of the plurality of images can be identified. The method further includes the step of performing a pre-processing step of cutting the images in the region of interest respectively for the plurality of images. The classifier processes the output of the pre-processing, and detects the types and positions of the objects contained in the multiple images respectively, and generates multiple detection data corresponding to the multiple images. The method further includes a post-processing step of receiving the plurality of detection data and determining whether the position of the object included in each detection data is included in the interest region inherent in the original image.

Images