Strobe camera, automobile, vehicle lamp, object recognition system, calculation processing device, object recognition method, image display system, inspection method, camera device, image processing device

Abstract

The strobe camera (20) divides the depth direction into a plurality of ranges and generates a plurality of slice images corresponding to the plurality of ranges. The lighting device (22) irradiates the detection light (L1). The controller (26) controls the irradiation of the detection light (L1) by the lighting device (22) and the timing of the shooting by the first image sensor (24A) and the second image sensor (24B). The controller (26) controls so that the first image sensor (24A) and the second image sensor (24B) respectively receive reflected light of different ranges corresponding to the irradiation of the detection light by the lighting device (22).

Description

Technical Field

[0001] The present invention relates to sensing and object recognition. Background Art

[0002] An object recognition system is used to sense the position and type of objects around the vehicle in order to perform autonomous driving or automatic control of the light distribution of the headlights. The object recognition system includes a sensor and a processing unit for analyzing the sensor output. The sensor is selected from cameras, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), millimeter wave radar, ultrasonic sonar, etc., taking into account the purpose, required accuracy, and cost.

[0003] Depth information is not available from ordinary SLR cameras, so it is difficult to separate multiple objects at different distances when they overlap.

[0004] A TOF camera is a camera that can obtain depth information. A TOF (Time Of Flight) camera emits infrared light through a light emitting device, measures the flight time of the reflected light returning to the image sensor, and obtains an image that converts the flight time into distance information.

[0005] An active sensor (hereinafter referred to as a gated camera in this specification) is proposed to replace the TOF camera (Patent Documents 1 and 2). The gated camera divides the shooting range into multiple ranges and performs multiple imaging by changing the exposure timing and exposure time for each range. Therefore, a slice image is obtained for each range of the object, and each slice image only contains the object contained in the corresponding range.

[0006] [Prior art literature]

[0007] [Patent Document]

[0008] Patent Document 1: Japanese Patent Application Publication No. 2009-257981

[0009] Patent Document 2: International Publication WO2017 / 110417A1 Summary of the invention

[0010] [Technical problem to be solved by the invention]

[0011] As a result of studying the existing strobe camera, the inventor finally realized the following problem. The existing strobe includes an illumination device for irradiating the detection light and a single camera group for measuring the reflected light. When a range is set as a measurement object (referred to as the object range), only the reflected light from the object range is measured by the camera, and the reflected light from the objects included in other non-object ranges is not measured and discarded. In other words, the utilization efficiency of the detection light is not high.

[0012] An aspect of the present invention is obtained in view of the related situation, and one of the exemplary purposes of the aspect is to provide a gated camera that improves the utilization efficiency of detection light or increases the frame rate.

[0013] [Methods for solving technical problems]

[0014] 1. One embodiment of the present invention relates to a gated camera that divides the depth direction into a plurality of ranges and generates a plurality of slice images corresponding to the plurality of ranges. The gated camera includes: an illumination device that irradiates detection light; a first image sensor; a second image sensor; and a controller that controls the timing of the illumination device irradiating the detection light and the first image sensor and the second image sensor taking pictures. The controller controls so that the first image sensor and the second image sensor receive reflected light from different ranges, respectively, corresponding to one illumination of the detection light by the illumination device.

[0015] 2. One embodiment of the present invention relates to an object recognition system. The object recognition system includes: a visible light camera that generates a visible light image; a strobe camera that divides the depth direction into multiple ranges and generates multiple infrared images corresponding to the multiple ranges; and a processing device that processes the visible light image and the multiple infrared images. The processing device detects independent objects in each of the multiple infrared images, applies the object contours determined for the multiple infrared images to the visible light image, and separates the multiple object images contained in the visible light image.

[0016] 3. One embodiment of the present invention relates to an object recognition system. The object recognition system includes: a strobe camera that divides the depth direction into a plurality of ranges, and for each range, changes the time difference between light emission and exposure to perform photography, thereby generating a plurality of slice images corresponding to the plurality of ranges; and an operation processing device that specifies a plurality of first correction characteristics corresponding to the plurality of ranges, corrects each of the plurality of slice images using the corresponding first correction characteristics, and determines the type of the object based on the corrected plurality of slice images.

[0017] Another solution of the present invention is a computing device. The computing device is used together with a strobing camera to constitute an computing device of an object recognition system. The strobing camera divides the depth direction into a plurality of ranges, and for each range, changes the time difference between light emission and exposure to perform shooting, and generates a plurality of slice images corresponding to the plurality of ranges. The computing device includes: a first correction unit having a plurality of first correction characteristics corresponding to the plurality of ranges, and correcting each of the plurality of slice images using the corresponding first correction characteristics; and a classifier, which determines the type of the object based on the plurality of corrected slice images.

[0018] 4. One embodiment of the present invention relates to an image display system. The image display system includes: a strobe camera that divides the depth direction into a plurality of ranges, and for each range, changes the time difference between light emission and exposure to perform photographing, thereby generating a plurality of slice images corresponding to the plurality of ranges; and an operation processing device that synthesizes the plurality of slice images to generate a synthesized image for display on a display.

[0019] Another embodiment of the present invention is a processing device. The processing device is used together with a strobe camera to form an image display system. The processing device includes: a synthesis unit that synthesizes a plurality of slice images to generate an image for display on a display; and an output unit that outputs the image generated by the synthesis unit to the display.

[0020] 5. One aspect of the present invention relates to a gated camera that divides the depth direction into a plurality of ranges and generates a plurality of slice images corresponding to the plurality of ranges. The gated camera includes: an illumination device that irradiates detection light according to a light emission timing signal; an image sensor that captures reflected light according to a photographic timing signal; and a controller that generates a light emission timing signal and a photographic timing signal with a time difference that is respectively specified for each range. The gated camera is configured to be able to give an offset to the time difference between the light emission timing signal and the photographic timing signal during inspection.

[0021] Another embodiment of the present invention is still a strobe camera. The strobe camera includes: an illumination device that irradiates detection light according to a light emission timing signal; an image sensor that captures reflected light according to a photographic timing signal; and a controller that generates a light emission timing signal and a photographic timing signal with a time difference specified for each range. During inspection, the light emission timing signal can be output to an external reference light source via an external delay unit.

[0022] Another embodiment of the present invention is a strobe camera. The strobe camera includes: an illumination device that irradiates detection light according to a light emission timing signal; an image sensor that captures reflected light according to a photographic timing signal; and a controller that generates a light emission timing signal and a photographic timing signal with a time difference specified for each range. During inspection, the photographic timing signal can be output to an external image sensor via an external delay unit.

[0023] 6. One embodiment of the present invention relates to an imaging device. The imaging device includes: a strobe camera that divides the depth direction into a plurality of ranges, changes the time difference between light emission and exposure for each range and captures the image, thereby generating a plurality of slice images corresponding to the plurality of ranges; and an image processing device that synthesizes the plurality of slice images to generate a synthesized image. The image processing device detects a non-existent area where an object does not exist for each of the plurality of slice images, performs noise reduction processing on the non-existent area, and synthesizes the plurality of slice images after noise reduction.

[0024] 7. Another embodiment of the present invention is an image processing device. The image processing device is configured to perform: for each of a plurality of slice images taken by a strobe camera, a process of detecting a non-existent area where an object does not exist; a process of reducing noise in the non-existent area; and a process of synthesizing the plurality of slice images after noise reduction.

[0025] 8. Another embodiment of the present invention is an object recognition system, comprising: a visible light camera for generating a visible light image, a selection camera for dividing the depth direction into a plurality of ranges and generating a plurality of infrared images corresponding to the plurality of ranges, and a processing unit for processing the visible light image and the plurality of infrared images; the processing unit (i) detects an independent object in each of the plurality of infrared images and extracts the contour of the object, and (ii) applies the contour to the visible light image to separate the plurality of object images contained in the visible light image.

[0026] 9. In the object recognition system as described in Scheme 8, in each infrared image, the processing device determines an independent single area and a plurality of continuous areas as independent objects.

[0027] 10. In the object recognition system as described in Option 8 or 9, when the accuracy of object recognition based only on the visible light image is low, the gating camera is activated.

[0028] 11. In the object recognition system as described in Option 8 or 9, the gating camera operates at a lower frame rate than the visible light camera.

[0029] 12. In the object recognition system according to any one of schemes 8 to 11, in the arithmetic processing device, for an object whose contour has been extracted once, the contour is continuously used and further extraction of the contour is stopped.

[0030] 13. The object recognition system according to any one of schemes 8 to 12 is mounted on a vehicle, wherein the visible light camera and the strobe camera are built-in close to one of the left and right headlights.

[0031] 14. One embodiment of the present invention is a car, comprising the object recognition system according to any one of embodiments 8 to 13.

[0032] 15. One embodiment of the present invention is a vehicle lamp, comprising the object recognition system according to any one of embodiments 8 to 13.

[0033] 16. One embodiment of the present invention is a computing processing device, which is used together with a visible light camera for generating visible light images and a gating camera to constitute an computing processing device of an object recognition system. The gating camera divides the depth direction into a plurality of ranges, generates a plurality of infrared images corresponding to the plurality of ranges, detects independent objects in each of the plurality of infrared images, applies the contour of each object to the visible light image, and segments the plurality of object images contained in the visible light image.

[0034] 17. One embodiment of the present invention is an object recognition method, comprising: a step of generating a visible light image by means of a visible light camera; a step of dividing the depth direction into a plurality of ranges by selecting a camera, and generating a plurality of infrared images corresponding to the plurality of ranges; a step of detecting independent objects in each of the plurality of infrared images, and generating contour data of each object; and a step of applying the contour data to the visible light image, and segmenting the plurality of object images contained in the visible light image.

[0035] 18. One scheme of the present invention is an object recognition system, comprising: a selection camera, which divides the depth direction into a plurality of ranges, and for each range, changes the time difference between light emission and exposure to perform shooting, thereby generating a plurality of slice images corresponding to the plurality of ranges; and an operation processing device, which specifies a plurality of first correction characteristics corresponding to the plurality of ranges, uses the corresponding first correction characteristics to correct each of the plurality of slice images, and determines the type of the object based on the corrected plurality of slice images.

[0036] 19. In the object recognition system as described in Scheme 18, the first correction characteristic corresponding to each range changes according to the measurement environment.

[0037] 20. In the object recognition system as described in Scheme 18 or 19, the processing unit specifies a plurality of second correction characteristics corresponding to the plurality of ranges, uses the corresponding second correction characteristics to correct each of the plurality of slice images, and synthesizes the corrected plurality of slice images.

[0038] 21. The object recognition system as described in Option 20 displays an image synthesized from the corrected multiple slice images on a display.

[0039] 22. One embodiment of the present invention is a car, comprising: an object recognition system as described in any one of embodiments 18 to 21.

[0040] 23. One embodiment of the present invention is a vehicle lamp, comprising: an object recognition system as described in any one of embodiments 18 to 21.

[0041] 24. One embodiment of the present invention is an operation processing device, which is used together with a gating camera to constitute an operation processing device of an object recognition system. The gating camera divides the depth direction into multiple ranges, and for each range, changes the time difference between light emission and exposure and takes pictures to generate multiple slice images corresponding to the multiple ranges. The operation processing device includes: a first correction unit, which specifies multiple first correction characteristics corresponding to the multiple ranges, and uses the corresponding first correction characteristics to correct each of the multiple slice images; and a classifier, which determines the type of the object based on the corrected multiple slice images.

[0042] 25. The operation processing device as described in Scheme 24 also includes: a second correction unit, which specifies a plurality of second correction characteristics corresponding to the plurality of ranges, and uses the corresponding second correction characteristics to correct each of the plurality of slice images; a synthesis unit, which synthesizes the plurality of slice images corrected by the second correction unit; and an output unit, which outputs the image generated by the synthesis unit to a display.

[0043] 26. One scheme of the present invention is an image display system, which selects a camera to divide the depth direction into a plurality of ranges, and for each range, changes the time difference between light emission and exposure to perform shooting, thereby generating a plurality of slice images corresponding to the plurality of ranges; and an operation processing device to synthesize the plurality of slice images to generate a synthesized image for display on a display.

[0044] 27. In the image display system as described in Scheme 26, in the arithmetic processing device, a plurality of correction characteristics corresponding to a plurality of ranges are specified, each of the plurality of slice images is corrected using the corresponding correction characteristics, and the corrected plurality of slice images are synthesized.

[0045] 28. In the image display system as described in Scheme 27, the correction characteristics corresponding to each range vary depending on the measurement environment.

[0046] 29. One embodiment of the present invention is a car, comprising: an image display system as described in any one of embodiments 26 to 28.

[0047] 30. One embodiment of the present invention is an operation processing device, which is used together with a gating camera to constitute an operation processing device of an image display system. The gating camera divides the depth direction into a plurality of ranges, and for each range, changes the time difference between light emission and exposure to perform shooting, thereby generating a plurality of slice images corresponding to the plurality of ranges. The operation processing device includes: a synthesis unit, which synthesizes the plurality of slice images to generate an image for display on a display; and an output unit, which outputs the image generated by the synthesis unit to the display.

[0048] 31. The operation processing device as described in Scheme 30 further includes a correction unit, which specifies multiple correction characteristics corresponding to the multiple ranges, uses the corresponding correction characteristics to correct each of the multiple slice images, and the synthesis unit synthesizes the multiple slice images corrected by the second correction unit.

[0049] 32. One embodiment of the present invention is a gating camera, which divides the depth direction into multiple ranges and generates multiple slice images corresponding to the multiple ranges. The gating camera comprises: an illumination device for irradiating detection light according to a light emission timing signal, an image sensor for photographing reflected light according to a photographing timing signal, and a controller for generating the light emission timing signal and the photographing timing signal with a time difference respectively specified for each range; the gating camera is configured to assign an offset to the time difference between the light emission timing signal and the photographing timing signal during the inspection process.

[0050] 33. In the gating camera as described in Option 32, an external delay unit can be connected to the path of the light-emitting timing signal.

[0051] 34. The gating camera as described in Option 32 also includes a delay unit, which is inserted into the path of the light-emitting timing signal during the inspection process.

[0052] 35. In the gating camera as described in Option 32, an external delay unit can be connected to the path of the photography timing signal.

[0053] 36. The gating camera as described in Option 32 also includes a delay unit, which is inserted into the path of the photography timing signal during the inspection process.

[0054] 37. One embodiment of the present invention is a gating camera, which divides the depth direction into multiple ranges and generates multiple slice images corresponding to the multiple ranges. The gating camera comprises: an illumination device for irradiating detection light according to a light emission timing signal, an image sensor for photographing reflected light according to a photographing timing signal, and a controller for generating the light emission timing signal and the photographing timing signal with a time difference respectively specified for each range; in the inspection process, the light emission timing signal can be output to an external reference light source via an external delay unit.

[0055] 38. One embodiment of the present invention is a gating camera, which divides the depth direction into multiple ranges and generates multiple slice images corresponding to the multiple ranges. The gating camera comprises: an illumination device for irradiating detection light according to a light emission timing signal, an image sensor for photographing reflected light according to a photographic timing signal, and a controller for generating the light emission timing signal and the photographic timing signal with a time difference specified for each range; during the inspection process, the photographic timing signal can be output to an external image sensor via an external delay unit.

[0056] 39. One embodiment of the present invention is an inspection method, which is an inspection method using a gated camera that divides the depth direction into multiple ranges and generates multiple slice images corresponding to the multiple ranges, the gated camera comprising: an illumination device that irradiates detection light according to a light emission timing signal, an image sensor that captures reflected light according to a photographic timing signal, and a controller that generates the light emission timing signal and the photographic timing signal with a time difference specified for each range; the inspection method comprises: a step of configuring a reflector at the i-th range, where i is a natural number, a step of actuating the controller in a state where the j-th range can be captured, where j>i, and a step of applying a predetermined delay to the light emission timing signal.

[0057] 40. One embodiment of the present invention is an inspection method, which is an inspection method using a gating camera that divides the depth direction into multiple ranges and generates multiple slice images corresponding to the multiple ranges, the gating camera comprising: an illumination device that irradiates detection light according to a light emission timing signal, an image sensor that captures reflected light according to a photographic timing signal, and a controller that generates the light emission timing signal and the photographic timing signal with a time difference specified for each range; the inspection method comprises: a step of configuring a reflector at the jth range, where j is a natural number, a step of actuating the controller in a state where the i-th range can be captured, wherein i<j, and a step of applying a predetermined delay to the photographic timing signal.

[0058] 41. One embodiment of the present invention is an imaging device, comprising: a selection camera, which divides the depth direction into a plurality of ranges, and for each range, changes the time difference between light emission and exposure to perform shooting, thereby generating a plurality of slice images corresponding to the plurality of ranges, and an image processing device, which synthesizes the plurality of slice images to generate a synthesized image; the image processing device detects a non-existent area where an object does not exist for each of the plurality of slice images, performs noise reduction processing on the non-existent area, and synthesizes the plurality of slice images after noise reduction.

[0059] 42. In the imaging device as described in solution 41, when the image processing device detects a presence area where an object exists in a certain slice image, it determines the area of ​​other slice images overlapping with the object as the non-existence area.

[0060] 43. In the imaging device as described in Scheme 41 or 42, the image processing device sets the pixel values ​​of multiple pixels included in the non-existent area to zero.

[0061] 44. In the imaging device according to any one of schemes 41 to 43, when there are a plurality of pixels whose pixel values ​​are greater than a threshold value in succession, the image processing device determines that the area including these pixels is an area where an object exists.

[0062] 45. One embodiment of the present invention is a car, comprising: a camera device as described in any one of embodiments 41 to 44; and a classifier for determining the type of an object contained in a composite image generated by the camera device.

[0063] 46. ​​One embodiment of the present invention is a car, comprising: a camera device as described in any one of embodiments 41 to 44; and a display device for displaying a composite image generated by the camera device.

[0064] 47. One embodiment of the present invention is an image processing device, which is configured to perform the following processing: detecting a non-existent area where an object does not exist in each of a plurality of slice images captured by a selected camera; performing noise reduction processing on the non-existent area; and synthesizing the plurality of slice images after noise reduction.

[0065] 48. In the image processing device described in the above scheme 47, when a presence area where an object exists is detected in a certain slice image, the area of ​​other slice images overlapping with the object is determined as the non-existence area.

[0066] [Effects of the invention]

[0067] According to one aspect of the present invention, the utilization efficiency of detection light can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a block diagram of the object recognition system according to the first embodiment.

[0069] Figure 2 This is a diagram for explaining the basic operation of the strobe camera.

[0070] Figure 3 (a) and (b) are diagrams illustrating slice images obtained by a gated camera having one camera.

[0071] Figure 4 Yes Description Figure 1 Figure 4 shows the action of the strobe camera.

[0072] Figure 5(a) and (b) are diagrams illustrating slice images obtained by gating the camera.

[0073] Figure 6 (a) is a diagram showing the correspondence between the range and the camera of Example 1-1, Figure 6 (b) corresponds to Figure 6 (a) Timing diagram of the allocation.

[0074] Figure 7 (a) is a diagram showing the correspondence between the range and the camera of Example 1-2, Figure 7 (b) shows the corresponding Figure 7 (a) Timing diagram of the allocation.

[0075] Figure 8 It is a diagram showing a strobe camera according to Embodiment 1-2.

[0076] Fig. 9 (a) and (b) are explanations Figure 8 Figure 4 shows the action of the strobe camera.

[0077] Fig.10 (a) to (c) are diagrams showing the range distribution of Examples 1-3.

[0078] Fig.11 (a) and (b) are diagrams showing a car equipped with a strobe camera.

[0079] Fig.12 : is a block diagram showing a vehicle lamp including an object detection system.

[0080] Fig.13 is a diagram showing an example of a scene in which a plurality of objects exist.

[0081] Fig.14 (a) shows the image captured by a visible light camera Fig.13 The visible light image obtained for the scene, Fig.14 (b) shows the image captured by an infrared camera Fig.13 Figure 1 shows an infrared image of a scene.

[0082] Fig.15 This is a block diagram of an object recognition system according to embodiment 2.

[0083] Fig.16 This is a diagram for explaining the basic operation of the strobe camera.

[0084] Fig.17 (a) and (b) are diagrams illustrating images obtained by strobing the camera.

[0085] Fig.18It is a functional block diagram of a processing device according to an embodiment.

[0086] Fig.19 (a) and (b) are diagrams for explaining the determination process of an independent object based on one infrared image.

[0087] Fig. 20 (a) to (c) are diagrams for explaining the operation of the object recognition system.

[0088] Fig.21 This is a functional block diagram of a processing device according to variation 2-1.

[0089] Fig. 22 (a) and (b) are diagrams showing a car equipped with a strobe camera.

[0090] Fig.23 : is a block diagram showing a vehicle lamp including an object detection system.

[0091] Fig.24 This is a block diagram of an object recognition system of implementation mode 3-1.

[0092] Fig.25 A diagram for explaining the operation of the strobe camera.

[0093] Fig.26 (a) and (b) are diagrams illustrating images obtained by a gated camera.

[0094] Fig. 27 (a) to (d) are diagrams showing several examples of correction characteristics.

[0095] Fig.28 (a) and (b) are diagrams for explaining photography performed by a strobe camera.

[0096] Fig.29 (a) and (b) are diagrams for explaining correction performed by the first correction unit.

[0097] Fig.30 This is a block diagram of an object recognition system of implementation mode 3-2.

[0098] Fig.31 This is a diagram showing an object recognition system according to embodiment 3-3.

[0099] Fig.32 (a) to (d) are diagrams illustrating image displays of an object recognition system.

[0100] Fig.33 This is a diagram showing an object recognition system according to Embodiment 3-4.

[0101] Fig.34 is a block diagram of a car with an object recognition system.

[0102] Fig.35 : is a block diagram showing a vehicle lamp including an object detection system.

[0103] Fig.36 (a) and (b) are diagrams for explaining the characteristics of a conventional active sensor.

[0104] Fig.37 It is a block diagram of an image display system according to embodiment 4-1.

[0105] Fig.38 A diagram for explaining the operation of the strobe camera.

[0106] Fig.39 (a) and (b) are diagrams illustrating images obtained by strobing the camera.

[0107] Fig.40 (a) and (b) are diagrams for explaining photography performed by a strobe camera.

[0108] Fig.41 (a) shows the synthesis Fig.40 (b) is a composite image obtained by combining multiple slice images. Fig.41 (b) is shown in Fig.40 (a) is an image obtained by using an existing infrared active sensor in the scene.

[0109] Fig.42 It is a block diagram of an image display system according to embodiment 4-2.

[0110] Fig.43 (a) to (d) are diagrams showing several examples of correction characteristics.

[0111] Fig.44 (a) and (b) are explanations Fig.42 FIG. 4 is a diagram of the processing of the correction part.

[0112] Fig.45 It is a block diagram of an image display system according to embodiment 4-3.

[0113] Fig.46 This is a block diagram of the selection camera of implementation mode 5-1.

[0114] Fig.47 This is a diagram for explaining the basic operation of the strobe camera.

[0115] Fig.48 (a) and (b) are diagrams illustrating slice images obtained by a gated camera.

[0116] Fig.493 is a diagram showing the layout of the gate camera and the reference reflector REF in the inspection process.

[0117] Fig.50 Yes Description Fig.46 Figure 1. Inspection of a gated camera.

[0118] Fig.51 It is shown Fig.46 FIG. 1 is a diagram of a first configuration example of a strobe camera.

[0119] Fig.52 It is shown Fig.46 FIG. 1 is a diagram of a second configuration example of a strobe camera.

[0120] Fig.53 This is a block diagram of the selection camera of implementation mode 5-2.

[0121] Fig.54 3 is a diagram showing the layout of the gate camera and the reference reflector REF in the inspection process.

[0122] Fig.55 Yes Description Fig.53 Figure 1. Inspection of a gated camera.

[0123] Fig.56 It is shown Fig.53 FIG. 1 is a diagram of a first configuration example of a strobe camera.

[0124] Fig.57 It is shown Fig.53 FIG. 1 is a diagram of a second configuration example of a strobe camera.

[0125] Fig.58 This is a diagram showing a strobe camera according to embodiment 5-3.

[0126] Fig.59 This is a diagram showing a strobe camera according to embodiment 5-4.

[0127] Fig.60 : is a block diagram showing a vehicle lamp including an object detection system.

[0128] Fig.61 (a) and (b) are diagrams for explaining problems related to image synthesis.

[0129] Fig.62 This is a diagram showing an imaging device according to embodiment 6-1.

[0130] Fig.63 A diagram for explaining the operation of the strobe camera.

[0131] Fig.64 (a) and (b) are diagrams illustrating images obtained by a gated camera.

[0132] Fig.65 This is a functional block diagram of an image processing device.

[0133] Fig.66 It is a stereoscopic image of the shooting scene.

[0134] Fig.67 (a) to (c) are diagrams for explaining synthesis processing by the image processing device.

[0135] Fig.68 (a) to (c) are diagrams showing other examples of the operation of the imaging device according to Embodiment 6-1.

[0136] Fig.69 (a) to (c) are diagrams illustrating the synthesis processing of embodiment 6-2.

[0137] Fig.70 This is a block diagram of an object recognition system including an imaging device.

[0138] Fig.71 This is a block diagram of a display system including an imaging device.

[0139] Fig.72 FIG. 1 is a diagram showing a car equipped with an object recognition system.

[0140] Fig.73 : is a block diagram showing a vehicle lamp including an object detection system. DETAILED DESCRIPTION

[0141] (Overview of Implementation Method 1)

[0142] 1. One embodiment disclosed in this specification relates to a gating camera. The gating camera divides the depth direction into multiple ranges and generates multiple slice images corresponding to the multiple ranges. The gating camera includes: an illumination device for irradiating detection light; a first image sensor; a second image sensor; and a controller for controlling the irradiation of the detection light of the illumination device and the timing of the first image sensor and the second image sensor. The controller controls so that the first image sensor and the second image sensor receive reflected light from different ranges respectively, corresponding to the irradiation of the detection light of the illumination device.

[0143] In the conventional strobe camera with a single camera, when a certain range is taken as the object, the light reflected by the object in other ranges is not used, so the utilization efficiency of the detection light is low. In contrast, according to the strobe camera, multiple slice images corresponding to multiple ranges can be obtained in one irradiation of the detection light, thereby improving the utilization efficiency of the detection light.

[0144] Alternatively, the first image sensor is assigned to a relatively short range, and the second image sensor is assigned to a relatively long range. The field of view of the first image sensor may be wider than the field of view of the second image sensor. Thus, the difference in the size of the object image can be reduced by using the slice image of the longer range and the slice image of the shorter range.

[0145] Next, Embodiment 1 will be described with reference to the drawings.

[0146] Figure 1 This is a block diagram of an object recognition system 10 according to Embodiment 1. The object recognition system 10 is mounted on a vehicle such as an automobile or a motorcycle, and determines the type (also referred to as category or type) of an object OBJ existing around the vehicle.

[0147] The object recognition system 10 mainly includes a strobe camera 20 and a processing device 40. The strobe camera 20 includes an illumination device 22, a first image sensor 24A, a second image sensor 24B, and a controller 26. The strobe camera 20 captures the depth direction by dividing the depth direction into a plurality of N (N≧2) ranges RNG1 to RNG N Adjacent measuring ranges may also overlap in the depth direction at their boundaries.

[0148] The lighting device 22 irradiates the detection light L1 toward the front of the vehicle in synchronization with the light emission timing signal S1 supplied from the controller 26. The detection light L1 is preferably infrared light, but is not limited thereto, and may be visible light or ultraviolet light having a predetermined wavelength.

[0149] The first image sensor 24A and the second image sensor 24B can perform exposure control in synchronization with the imaging timing signals S2A and S2B supplied by the controller 26, and are configured to generate slice images IMGA and IMGB. The first image sensor 24A and the second image sensor 24B have sensitivity at the same wavelength as the detection light L1, and capture reflected light (return light) L2 reflected by the object OBJ.

[0150] The controller 26 controls the irradiation of the detection light L1 by the lighting device 22 and the timing of photographing by the first image sensor 24A and the second image sensor 24B.

[0151] Figure 2 2 is a diagram for explaining the basic operation of the strobe camera 20. Here, for easy understanding, attention is paid to the operation of one (24A) of the first image sensor 24A and the second image sensor 24B. Figure 2 The measurement of the i-th range RNG is shown in iThe lighting device 22 is synchronized with the light-emitting timing signal S1 and emits light during the light-emitting period τ1 between time t0 and t1. The light diagram is shown at the top with the horizontal axis representing time and the vertical axis representing distance. i The distance to the boundary in the close direction is set to d MINi , will reach the range RNG i The distance from the depth boundary is set to d MAXi .

[0152] At a certain moment, the light emitted by the lighting device 22 reaches a distance d MINi , the round trip time T until the reflected light returns to the first image sensor 24A MINi for

[0153] T MINi =2×d MINi / c.

[0154] C is the speed of light.

[0155] Similarly, at a certain moment, the light emitted by the lighting device 22 reaches the distance d MAXi , the round trip time T until the reflected light returns to the first image sensor 24A MAXi for

[0156] T MAXi =2×d MAXi / c.

[0157] When you need to shoot only the range RNG i When the object OBJ is included in the image, the controller 26 generates a photographing timing signal S2A so that at time t2=t0+T MINi Start exposure, and at time t3 = t1 + T MAXi End the exposure. This is a one-time exposure action.

[0158] When shooting the i-th range RNG i At this time, the controller 26 repeats the above exposure action multiple times according to the specified cycle τ2.

[0159] Figure 3 (a) and (b) are diagrams for explaining slice images obtained by a gated camera 20 having one camera. Figure 3 In the example (a), the object (pedestrian) OBJ2 exists in the range RNG2, and the object (vehicle) OBJ3 exists in the range RNG3. Figure 3 In (b), it is shown that Figure 3When the slice image IMG1 is captured, the image sensor is exposed only by the reflected light from the range RNG1, so that no object image is captured in the slice image IMG1.

[0160] When capturing slice image IMG2, the image sensor is exposed only by the reflected light from range RNG2, so that only object image OBJ2 is captured in slice image IMG2. Similarly, when capturing slice image IMG3, the image sensor is exposed only by the reflected light from range RNG3, so that only object image OBJ3 is captured in slice image IMG3. In this way, by selecting camera 20, objects can be captured separately for each range.

[0161] return Figure 1 In this embodiment, the first image sensor 24A and the second image sensor 24B are assigned to different measurement ranges. More specifically, the controller 26 controls so that in response to the illumination of the detection light L1 by the lighting device 22, the first image sensor 24A and the second image sensor 24B receive reflected light from different measurement ranges.

[0162] By strobing the camera 20, multiple ranges RNG1 to RNG are generated. N Corresponding multiple slice images IMG1~IMG N In the i-th slice image IMG i In the RNG, only the corresponding range is reflected i The objects contained.

[0163] The calculation processing device 40 is configured to obtain the plurality of ranges RNG1 to RNG2 obtained by the strobe camera 20. N Corresponding multiple slice images IMG1~IMG N , capable of identifying the type of object. The operation processing device 40 includes a classifier 42, which is implemented based on a prediction model generated by machine learning. The algorithm of the classifier 42 is not particularly limited, but YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Region-based Convolutional Neural Network), SPPnet (Spatial Pyramid Pooling), Faster R-CNN, DSSD (Deconvolution-SSD), Mask R-CNN, etc. can be used, or an algorithm to be developed in the future can be used.

[0164] The operation processing device 40 can be implemented by a combination of a processor (hardware) such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a microcomputer, and a software program executed by the processor (hardware). The operation processing device 40 can also be a combination of multiple processors. That is, the operation processing device 40 can also be composed of hardware only.

[0165] The above is the configuration of the object recognition system 10 including the strobe camera 20. Next, the operation of the strobe camera 20 will be described. Figure 4 Yes Description Figure 1 FIG. 2 is a diagram illustrating the operation of the strobe camera 20 .

[0166] The first image sensor 24A is assigned to the relatively close i-th range RNG i , the relatively far second image sensor 24B is assigned to the jth range RNG j (j>i). The lighting device 22 is synchronized with the light-emission timing signal S1 and emits light during the light-emission period τ1 between time t0 and t1. The top part shows a graph of light with the horizontal axis as time and the vertical axis as distance d. The camera 20 is switched to the range RNG. i The distance from the near side boundary is denoted as d MINi , set the strobe camera 20 to the range RNG i The distance from the far side boundary is denoted as d MAXi Similarly, the range RNG j The distance from the near side boundary is denoted as d MINj will be from the gate camera 20 to the range RNG i The distance from the deep side boundary is denoted as d MAXj .

[0167] To shoot the range RNG i Objects contained in OBJ i , generating a photographic timing signal S2A so that at time t2 = t0 + T MINi The exposure of the first image sensor 24A begins at time t3 = t1 + T MAXi End exposure.

[0168] In addition, using the same probe light, in order to capture the range RNG j Objects contained in OBJ j , generating a photographic timing signal S2B so that at time t4 = t0 + T MINj The exposure of the second image sensor 24B begins at time t5 = t1 + T MAXj End exposure.

[0169] T MINj =2×d MINj / c

[0170] T MAXj =2×d MAXj / c

[0171] Figure 5 (a) and (b) are diagrams for explaining slice images obtained by the gated camera 20. Figure 5 In the example (a), the object (pedestrian) OBJ i Existence in Quantum RNG i In the object (vehicle) OBJ j Exists in Quantum RNG j In. Figure 5 In (b), it is shown that Figure 5 In the case of (a), the two slice images IMG obtained by the first image sensor 24A and the second image sensor 24B are i 、IMG j In addition, the first image sensor 24A and the second image sensor 24B have the same field of view, and therefore, reflect the distant object OBJ less. j .

[0172] The above is the operation of the gate camera 20. Next, its advantages are described. In the conventional gate camera with a single camera, when a certain range is taken as the object, the light reflected by the object of other ranges is not used, so it can be said that the utilization efficiency of the detection light is low. In contrast, according to the gate camera 20 of this embodiment, multiple slice images corresponding to multiple ranges can be obtained in one irradiation of the detection light L1. That is, the utilization efficiency of the detection light can be improved.

[0173] Next, allocation of a plurality of ranges to the first image sensor 24A and the second image sensor 24B will be described based on several embodiments.

[0174] (Example 1-1)

[0175] For simplicity, all range RNG i The length d in the depth direction of (i=1~N) MAXi -d MINi The lengths are equal, and each range RNG is generated i The number of exposures required for the slice images is also equal.

[0176] Figure 6(a) is a diagram showing the correspondence between the ranges and the cameras of Example 1-1. In the diagram, A shows the first image sensor and B shows the second image sensor. Although not limited to this, in this example, the ranges are divided into N=6.

[0177] In Example 1-1, the odd-numbered ranges RNG1, RNG3, and RNG5 are allocated to the first image sensor 24A, and the even-numbered ranges RNG2, RNG4, and RNG6 are allocated to the second image sensor 24B.

[0178] Figure 6 (b) is related to Figure 6 The timing diagram corresponding to the allocation of (a). The combination of K times of light emission of the lighting device 22 and K times of exposure of the first image sensor 24A and the second image sensor 24B required to generate one slice image is called one time slot. Figure 6 In the example (b), one time slot includes one light emission and one exposure, but one time slot may include multiple light emissions and exposures. In one camera, six time slots are required for shooting with N=6 ranges, but in the worst case of two cameras, three time slots are required.

[0179] In the first to third time slots, the first image sensor 24A captures the three ranges RNG1, RNG3, and RNG5 assigned to it in a time-division manner. Similarly, the second image sensor 24B captures the three ranges RNG2, RNG4, and RNG6 assigned to it in a time-division manner.

[0180] (Example 1-2)

[0181] Figure 7 (a) is a diagram showing the correspondence between the range and the camera of Example 1-2. In the figure, A represents the first image sensor and B represents the second image sensor. One of the two cameras 24A and 24B is assigned to a plurality of relatively close ranges, and the other 24B is assigned to a plurality of relatively far ranges. RNG1 to RNG3 are assigned to the first image sensor 24A, and RNG4 to RNG6 are assigned to the second image sensor 24B.

[0182] Figure 7 (b) is related to Figure 7The timing diagram corresponding to the allocation of (a). In this example, 1 time slot includes 1 light emission and 1 exposure, but it is not limited to this. 1 time slot can include multiple light emissions and exposures. The shooting of N=6 ranges is completed through 3 time slots. In the first time slot to the third time slot, the first image sensor 24A shoots the 3 ranges RNG1~RNG3 assigned to itself in time division. Similarly, in the first time slot to the third time slot, the second image sensor 24B shoots the 3 ranges RNG4~RNG6 assigned to itself in time division.

[0183] The method of Example 1-2 has the following advantages over the method of Example 1-1: In the control of Example 1-2, when the output image of the first image sensor 24A and the output image of the second image sensor 24B are processed in parallel by two classifiers, the two classifiers can be implemented by an algorithm for close distance and an algorithm for long distance.

[0184] Preferably, in Embodiment 1-2, the field of view angles of the first image sensor 24A and the second image sensor 24B may be different. Figure 8 2 is a diagram showing the strobe camera 20 of Example 1-2. The first image sensor 24A assigned to the short range is configured to be able to capture a relatively wide field of view, and the second image sensor 24B assigned to the long range is configured to be able to capture a relatively narrow field of view.

[0185] For example, the first image sensor 24A and the second image sensor 24B may have the same size of image sensors, and a wide-angle lens may be combined with the first image sensor 24A, and a telephoto lens may be combined with the second image sensor 24B.

[0186] Alternatively, the focal lengths of the lenses of the first image sensor 24A and the second image sensor 24B may be the same, the size of the image sensor of the first image sensor 24A may be relatively increased, and the size of the image sensor of the second image sensor 24B may be reduced, thereby changing the field of view. Alternatively, the first image sensor 24A and the second image sensor 24B may have different sensor sizes and focal lengths. Different shadows are marked on the range captured by the first image sensor 24A and the range captured by the second image sensor 24B. The above is the configuration of the strobe camera 20 of Embodiment 1-2.

[0187] Fig. 9 (a) and (b) are explanations Figure 8 FIG. 2 is a diagram illustrating the operation of the strobe camera 20 .

[0188] For comparison, refer to Figure 5 . Use a camera with the same field of view to capture the near-range RNG i and long range RNG j When Figure 5 As shown in (b), the long range RNG j Object OBJ j It was photographed relatively small.

[0189] return Fig. 9 According to Embodiment 1-2, the close range RNG is photographed by cameras with different field of view angles. i (i=1~3), long range RNG j (j=4~6), can capture far-range RNG j Object OBJ j This can improve the recognition rate of the object in the arithmetic processing device 40 at the subsequent stage of the strobe camera 20 .

[0190] (Example 1-3)

[0191] The inventors of the present invention conducted research on strobe cameras and eventually recognized the following technical problems.

[0192] In the strobe camera, the image sensor captures the reflected light of the detection light emitted by the light emitting device reflected by the object included in the range of the photographed object. That is, the farther the distance of the reflected light received by the image sensor from the object range is, the smaller the intensity is.

[0193] In the case of a conventional camera, the decrease in light intensity can be compensated by increasing the exposure time. However, in a gated camera, the exposure time is determined based on the distance from the gated camera to the front side boundary of the range of the subject and the distance from the gated camera to the back side boundary of the range of the subject, so the exposure time cannot be increased.

[0194] Therefore, in order to obtain a good image at a longer range, the longer the range, the more measures need to be taken, such as (i) increasing the intensity of the irradiated probe light and (ii) increasing the sensitivity of the image sensor.

[0195] In the case where such a countermeasure is not adopted or there are many insufficient examples, in this case, it is necessary to perform complex K for a longer range RNGj. j Times (K j ≧2) Combination of light emission and shooting (time slots mentioned above), synthesizing the result to obtain K j Slice images (or K multiple exposures) are taken to generate one image. That is, the longer the range, the more time slots are required to generate one slice image.

[0196] For easy understanding, it is set as K1=K2=1, K3=K4=2, K5=K6=3. At this time, 12 time slots are required to generate all slice images IMG1 to IMG6 of the six ranges RNG1 to RNG6. When the time of one time slot is recorded as Ts, in the existing gated camera with a single camera, it takes 12×Ts to process one frame of 12 time slots.

[0197] In Embodiment 1-3, a technique for shortening the time of one frame is provided.

[0198] Fig.10 (a) to (c) are diagrams showing the range allocation of Example 1-3. In Example 1-3, the total number of required time slots is allocated to the first image sensor 24A and the second image sensor 24B in half, and processing is performed in parallel. Fig.10 (a) As in Example 1-2, the time slots of the shorter ranges RNG1 to RNG4 are allocated to the first image sensor 24A, and the time slots of the longer ranges RNG5 and RNG6 are allocated to the second image sensor 24B.

[0199] Fig.10 (b) As in Example 1-1, the time slots of the odd-numbered ranges RNG1, RNG3, and RNG5 are allocated to the first image sensor 24A, and the time slots of the even-numbered ranges RNG2, RNG4, and RNG6 are allocated to the second image sensor 24B.

[0200] Fig.10 (c) Two cameras are assigned to different time slots of the same range.

[0201] According to Embodiment 1-3, by allocating the processing of 12 time slots to the two cameras 24A and 24B and processing them in parallel, the processing time of one frame can be shortened to 6×Ts, and the frame rate can be improved.

[0202] Embodiment 1 has been described above. Next, a modification example related to Embodiment 1 will be described.

[0203] (Variation 1-1)

[0204] The camera allocation for multiple ranges or multiple time slots described in Examples 1-1 to 1-3 is an example, but those skilled in the art will appreciate that there are various modifications, and the modifications are also included in the scope of the present invention.

[0205] (Variation 1-2)

[0206] In the embodiment, the strobe camera 20 including the two cameras 24A and 24B has been described, but the number of cameras may be three or more.

[0207] (Variation 1-3)

[0208] In the embodiment, slice images of different ranges are processed by the same classifier 42 , but the present invention is not limited thereto, and a different classifier may be used for each range.

[0209] (use)

[0210] Fig.11 (a) and (b) are diagrams showing a car 300 equipped with a strobe camera 20. Fig.11 (a) The automobile 300 includes headlights (lamps) 302L and 302R. Fig.11 The automobile 300 of (a) has a lighting device 22 disposed in the center of the vehicle, and a first image sensor 24A and a second image sensor 24B are built into the left and right headlights 302L and 302R. The position of the lighting device 22 is not particularly limited, but it can be disposed, for example, in the front bumper (i) or the front grille (ii), or it can be installed on the inner side of the cabin glass on the inner side of the front window (iii). In addition, the position of the controller 26 is also not particularly limited, and it can be disposed in the engine compartment, in the cabin, or built into the headlight.

[0211] Reference Fig.11 (b). One lighting device 22 includes a plurality of (for example, two) light sources 22A and 22B. The plurality of light sources 22A and 22B emit light at the same timing, and the light emitted from them forms one detection light. The plurality of light sources 22A and 22B are built into the left and right headlights 302L and 302R.

[0212] The first image sensor 24A and the second image sensor 24B may be built into one of the headlights 302L and 302R. Alternatively, the first image sensor 24A and the second image sensor 24B may be provided outside the headlights 302L and 302R, or may be provided near the lighting device 22, for example.

[0213] Fig.12 2 is a block diagram showing a vehicle lamp 200 having an object detection system 210. The vehicle lamp 200 and the vehicle-side ECU 304 together constitute a lamp system 310. The vehicle lamp 200 includes a light source 202, a lighting circuit 204, and an optical system 206. In addition, the vehicle lamp 200 is provided with an object detection system 210. The object detection system 210 corresponds to the above-mentioned object recognition system 10, and includes a strobe camera 20 and an operation processing device 40.

[0214] The information related to the object OBJ detected by the calculation processing device 40 can be used in the light distribution control of the vehicle lamp 200. Specifically, the lamp-side ECU 208 generates an appropriate light distribution pattern based on the information related to the type of the object OBJ and its position generated by the calculation processing device 40. The lighting circuit 204 and the optical system 206 work to obtain the light distribution pattern generated by the lamp-side ECU 208.

[0215] In addition, the information related to the object OBJ detected by the processing unit 40 can be sent to the vehicle-side ECU 304. The vehicle-side ECU can automatically drive the vehicle based on the information.

[0216] (Implementation Method 2)

[0217] First, the issues related to the second embodiment will be described.

[0218] Fig.13 OBJ1 is a diagram showing an example of a scene in which multiple objects exist. When viewed from sensor 2 in the direction of the arrow, two objects OBJ1 and OBJ2 exist at overlapping positions. The two objects OBJ1 and OBJ2 are the same color, and the upper and lower surfaces of object OBJ2 are made of different materials.

[0219] Fig.14 (a) shows the image captured by a visible light camera Fig.13 When two objects are the same color or have very similar colors, their outlines (boundaries) become blurred in the visible light image. Therefore, when processing the visible light image, the two objects OBJ1 and OBJ2 may be mistakenly identified as the same object.

[0220] Fig.14 (b) shows the image taken with an infrared camera Fig.13 Infrared images are obtained by taking a scene. Infrared cameras make the parts with different reflectivity to infrared light appear as outlines. Fig.14 As shown in (b), in the infrared image, a clear outline is generated at the boundary between the upper and lower parts of the object OBJ2, where the materials are different. Therefore, when performing image processing on the infrared image, one object OBJ2 may be mistakenly identified as two objects OBJ 2A ,OBJ 2B .

[0221] The invention related to the second embodiment is obtained in view of the related situation, and one of the exemplary purposes of one aspect thereof is to provide an object recognition system that can correctly separate a plurality of objects.

[0222] (Overview of Implementation Method 2)

[0223] One embodiment disclosed in this specification relates to an object recognition system. The object recognition system includes: a visible light camera that generates a visible light image; a strobe camera that divides the depth direction into multiple ranges and generates multiple infrared images corresponding to the multiple ranges; and a processing device that processes the visible light image and the multiple infrared images. The processing device detects independent objects in the multiple infrared images, extracts the contours of the independent objects, applies the contours to the visible light image, and separates the multiple object images contained in the visible light image.

[0224] By combining the visible light image obtained by the visible light camera and the multiple infrared images obtained by the strobe camera, the object image can be separated by considering the depth information in addition to the color information and reflectivity information, thereby making it possible to more accurately separate multiple objects. Thus, for example, when objects of the same color overlap front and back, they can be distinguished.

[0225] In each infrared image, the processing device can determine an independent region or a plurality of continuous regions as independent objects, thereby recognizing an object having parts with different materials as one object.

[0226] When the accuracy of object recognition using only visible light images is low, the strobe camera can be activated. By keeping it to the minimum necessary, it is possible to suppress an increase in the amount of computational processing.

[0227] The strobe camera may be operated at a frame rate lower than that of the visible light camera, thereby reducing the amount of calculations performed by the calculation processing device.

[0228] In the operation processing device, it is also possible to continue to use the outline and stop extracting the outline again for the object of the temporary extraction outline. Once the object can be separated, since it is easy to track its outline thereafter, the operation processing capacity can be reduced by stopping the strobing camera.

[0229] Next, Embodiment 2 will be described with reference to the drawings.

[0230] Fig.15 This is a block diagram of an object recognition system 10 according to Embodiment 2. This object recognition system 10 is mounted on a vehicle such as an automobile or a motorcycle, and determines the type (also referred to as category or type) of an object OBJ existing around the vehicle.

[0231] The object recognition system 10 mainly includes a visible light camera 12, a strobe camera 20 and a processing device 40. The visible light camera 12 is a color camera or a black and white camera, and generates a visible light image IMG_VS.

[0232] The strobe camera 20 includes an illumination device 22, an infrared camera 24, and a controller 26. The strobe camera 20 captures images by dividing the depth direction into a plurality of N (N≧2) ranges RNG1 to RNG N Adjacent measuring ranges may also overlap at their boundaries in the depth direction.

[0233] The lighting device 22 irradiates the detection light L1 toward the front of the vehicle in synchronization with the light emission timing signal S1 supplied from the controller 26. The detection light L1 is preferably infrared light.

[0234] The infrared camera 24 is configured to generate an infrared image IMG_IR by performing exposure control in synchronization with the imaging timing signal S2 supplied from the controller 26. The infrared camera 24 has sensitivity at the same wavelength as the detection light L1 and captures reflected light (return light) L2 reflected from the object OBJ.

[0235] The controller 26 controls the timing of irradiation of the detection light L1 by the lighting device 22 and imaging by the infrared camera 24 .

[0236] Fig.16 It is a diagram for explaining the basic operation of the strobe camera 20. Fig.16 The measurement of the i-th range RNG is shown in i The lighting device 22 is synchronized with the light-emitting timing signal S1 and emits light during the light-emitting period τ1 between time t0 and t1. The light diagram is shown at the top with the horizontal axis representing time and the vertical axis representing distance. i The distance from the near side boundary is denoted as d MINi , set the strobe camera 20 to the range RNG i The distance from the far side boundary is denoted as d MAXi .

[0237] At a certain moment, the light emitted by the lighting device 22 reaches a distance d MINi The round trip time T of the reflected light returning to the infrared camera 24 MINi for

[0238] T MINi =2×d MINi / c.

[0239] c is the speed of light.

[0240] Similarly, at a certain moment, the light emitted by the lighting device 22 reaches a distance d MAXi The round trip time T of the reflected light returning to the infrared camera 24 MAXi for

[0241] T MAXi =2×d MAXi / c.

[0242] When you need to shoot only the range RNG i When the object OBJ is included in the image, the controller 26 generates a photographing timing signal S2 so that at time t2=t0+T MINi Start exposure at time t3 = t1 + T MAXi End the exposure. This is a one-time exposure action.

[0243] Shoot the i-th range RNG i At this time, the controller 26 can repeat the above exposure action multiple times with a predetermined period τ2.

[0244] Fig.17 (a) and (b) are diagrams for explaining images obtained by the strobe camera 20. Fig.17 In the example of (a), the object (pedestrian) OBJ2 exists in the range RNG2, and the object (vehicle) OBJ3 exists in the range RNG3. Fig.17 (b) shows that Fig.17 The infrared images IMG_IR1 to IMG3 are obtained in the situation (a). When the infrared image IMG_IR1 is captured, the image sensor is exposed only by the reflected light from the range RNG1, so no object image is captured in the infrared image IMG_IR1.

[0245] When capturing infrared image IMG_IR2, the image sensor is exposed only by the reflected light from range RNG2, so only object image OBJ2 is captured in infrared image IMG_IR2. Similarly, when capturing infrared image IMG_IR3, the image sensor is exposed only by the reflected light from range RNG3, so only object image OBJ3 is captured in infrared image IMGIR3. According to such a gated camera 20, objects can be captured separately for each range.

[0246] return Fig.15 The processing device 40 receives the visible light image IMG_VS and a plurality of infrared images IMG_IR1 to IMG_IR2. N , which is configured to be able to recognize objects in front.

[0247] The operation processing device 40 may include a classifier 42 and a separation processing unit 50. The operation processing device 40 may be implemented by a combination of a processor (hardware) such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a microcomputer, and a software program executed by the processor (hardware). The operation processing device 40 may also be a combination of multiple processors. That is, the operation processing device 40 may also be composed of only hardware.

[0248] The classifier 42 is implemented based on the prediction model generated by machine learning to determine the type (category, type) of the object contained in the input image. The algorithm of the classifier 42 is not particularly limited, but YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Region-based Convolutional Neural Network), SPPnet (Spatial Pyramid Pooling), Faster R-CNN, DSSD (Deconvolution-SSD), Mask R-CNN, etc. can be used, or an algorithm to be developed in the future can be used.

[0249] A separation processing unit 50 is provided in front of the classifier 42. The separation processing unit 50 does not determine the type of the object, but separates the object image IMG_OBJ corresponding to the object OBJ from the visible light image IMG_VS, and the classifier 42 identifies the type of each object image IMG_OBJ.

[0250] Fig.18 4 is a functional block diagram of the operation processing device 40 of an embodiment. In this embodiment, the separation processing unit 50 includes a first processing unit 52 and a second processing unit 54. The first processing unit 52 receives a plurality of infrared images IMG_IR1 to IMG_IR N The first processing unit 52 refers to each infrared image IMG_IR i , an independent area and multiple continuous areas are judged as independent objects OBJ ij . Refers to the range surrounded by the edge. OBJ ij Represents infrared image IMG_IR i The jth object contained in . Moreover, the output represents an independent object OBJ ij EDGE ijIf, in the i-th infrared image IMG i Contains M independent objects OBJ i1 ~OBJ iM In the case of M contour data EDGE is generated i1 ~EDGE iM The so-called "area" refers to the range surrounded by the edge. Contour data EDGE ij The data shown has no relation to the color of the object.

[0251] Fig.19 (a) and (b) illustrate the infrared image IMG_IR i FIG. 5 is a diagram showing determination of an independent object, that is, processing by the first processing unit 52 . Fig.19 (a) Infrared image IMG_IR i It contains 5 areas A to E. In addition, it should be noted that an infrared image IMG_IR i Included in the same range RNG i middle.

[0252] Region A is independent, so it is judged as an independent object OBJ i1 In addition, regions B and C are continuous, so they are considered as an independent object OBJ. i2 Similarly, regions D and E are continuous, so they are combined to form an independent object OBJ. i3 .

[0253] Fig.19 (b) is a diagram showing contour data. In this example, three independent objects OBJ are detected. i1 ~OBJ i3 , thereby generating three contour data EDGE representing their respective contours and positions i1 ~EDGE i3 .

[0254] return Fig.18 The second processing unit 54 inputs the infrared images IMG_IR1 to IMG_IR N The second processing unit 54 applies the object's outline EDGE to the visible light image IMG_VS, and cuts out the K object images IMG_OBJ1 to IMG_OBJ2 contained in the visible light image IMG_VS. K .

[0255] The above is the configuration of the object recognition system 10. Next, its operation will be described.

[0256] Fig. 20(a) to (c) of FIG. 1 are diagrams for explaining the operation of the object recognition system 10. Here, it is assumed that the object recognition system 10 measures Fig.13 scene. Fig.13 The object OBJ1 exists in the first range RNG1, and the object OBJ2 exists in the second range RNG2.

[0257] Reference Fig. 20 (a) and (b) illustrate the processing of the first processing unit 52. Fig. 20 (a) shows an infrared image IMG_IR1 obtained by the strobe camera 20. The infrared image IMG_IR1 captures the object OBJ1. The first processing unit 52 of the separation processing unit 50 detects the outline of the object OBJ1 and generates the outline data EDGE 11 .

[0258] Fig. 20 (b) shows an infrared image IMG_IR2 obtained by the strobe camera 20. The infrared image IMG_IR2 captures the object OBJ2. The first processing unit 52 of the separation processing unit 50 detects the outline of the object OBJ2 and generates the outline data EDGE 12 The upper and lower materials of object OBJ2 are different, but it is recognized as a separate object.

[0259] Reference Fig. 20 (c) illustrates the processing of the second processing unit 54. Fig. 20 The left side of (c) shows the visible light image IMG_VS obtained by the visible light camera 12. As described above, the two objects OBJ1 and OBJ2 have the same color, so their boundary is blurred in the visible light image IMG_VS. Fig. 20 As shown in the center of (c), the contour data EDGE is applied to the visible light image IMG_VS 11 ,EDGE 21 Therefore, if Fig. 20 As shown on the right side of (c), two object images IMG_OBJ1 and IMG_OBJ2 can be cut out. The second processing unit 54 sequentially applies the contour data of the closer range to sequentially cut out from the closer object image IMG_OBJ.

[0260] The above is the operation of the object recognition system 10. According to the object recognition system 10, a plurality of objects that are difficult to distinguish using only a visible light camera can be distinguished.

[0261] In addition, in the process of using a general infrared camera, such as Figure 2 As shown in (b), there is a situation where one object OBJ2 having parts of different materials is mistakenly recognized as two objects OBJ 2A ,OBJ 2BOn the other hand, according to the present embodiment, it is also possible to recognize an object having parts made of different materials as one object.

[0262] The frame rate of the gate camera 20 will be described. N All multiple infrared images IMG_IR1~IMG_IR N As 1 frame.

[0263] 1. The frame rate of the strobe camera 20 may be the same as the frame rate of the visible light camera 12. In this case, the same processing is performed on all frames, which can stably improve the accuracy of object recognition.

[0264] 2. If the frame rates of the visible light camera 12 and the strobe camera 20 are equal, there is a problem that the amount of calculation of the calculation processing device 40 increases. Therefore, the frame rate of the strobe camera 20 may be set lower than the frame rate of the visible light camera 12. In addition, object recognition based only on the visible light image IMG_VS and object recognition based on the visible light image IMG_VS and the infrared image IMG_IR may be performed in a time-division manner.

[0265] In this case, the intermittently generated outline data EDGE may be caused to track the position of the subject in the visible light image IMG_VS using a so-called tracking technique.

[0266] 3. Normally, object recognition is performed based only on the visible light camera 12. If the recognition accuracy decreases, the strobe camera 20 is activated to start object recognition based on the visible light image IMG_VS and the infrared image IMG_IR. The decrease in recognition accuracy can be detected based on the decrease in the probability. Thereafter, if the recognition rate improves, the strobe camera 20 can be deactivated to return to object recognition based only on the visible light camera 12.

[0267] A variation example related to Implementation Example 2 will be described.

[0268] (Variation 2-1)

[0269] The configuration of the arithmetic processing device 40 is not limited to the above-described configuration. Fig.21 FIG. 2 is a functional block diagram of a processing device 40A according to a modification 2-1. In this modification, the separation processing unit 50 includes a first processing unit 52. The functions of the first processing unit 52 are similar to those of Fig.18 Similarly, the first processing unit 52 refers to each infrared image IMG_IR i , an independent area, and then multiple continuous areas are determined as independent objects OBJ ij Moreover, the output represents an independent object OBJ ijEDGE ij The visible light image IMG_VS and a set of outline data EDGE are input to the classifier 42. The classifier 42 detects the object image IMG_OBJ included in the visible light image IMG_VS and determines the type thereof, based on each set of outline data EDGE.

[0270] For example, the outline data EDGE may be selected in sequence and used as a mask, so that only the image within the outline is the target of object recognition.

[0271] (use)

[0272] Fig. 22 (a) and (b) are diagrams showing a car 300 equipped with a strobe camera 20. Fig. 22 (a) The automobile 300 includes headlights (lamps) 302L and 302R. Fig. 22 The automobile 300 of (a) has a lighting device 22 disposed in the center of the vehicle, and has a visible light camera 12 and an infrared camera 24 built into the left and right headlights 302L and 302R. When the parallax between the visible light camera 12 and the infrared camera 24 is large, the processing unit 40 may be subjected to parallax correction. The position of the lighting device 22 is not particularly limited, but may be disposed, for example, in the front bumper (i) or the front grille (ii), or may be installed on the inner side of the cabin glass on the inner side of the front window (iii). In addition, the position of the controller 26 is not particularly limited, and may be disposed in the engine compartment, in the cabin, or built into the headlight.

[0273] Reference Fig. 22 (b). One lighting device 22 includes a plurality of (for example, two) light sources 22A and 22B. The plurality of light sources 22A and 22B emit light at the same timing, and the light emitted by them forms a detection light. The plurality of light sources 22A and 22B are built into the left and right headlights 302L and 302R.

[0274] The visible light camera 12 and the infrared camera 24 may be built into one of the headlights 302L and 302R. In this case, parallax correction between the visible light camera 12 and the infrared camera 24 is not necessary, and the processing of the arithmetic processing device 40 can be reduced.

[0275] The visible light camera 12 and the infrared camera 24 may be disposed outside the headlights 302L and 302R, or may be disposed near the lighting device 22, for example.

[0276] Fig.232 is a block diagram showing a vehicle lamp 200 having an object detection system 210. The vehicle lamp 200 and the vehicle-side ECU 304 together constitute a lamp system 310. The vehicle lamp 200 includes a light source 202, a lighting circuit 204, and an optical system 206. In addition, the vehicle lamp 200 is provided with an object detection system 210. The object detection system 210 corresponds to the above-mentioned object recognition system 10, and includes a strobe camera 20 and an operation processing device 40.

[0277] The information related to the object OBJ detected by the calculation processing device 40 can be used in the light distribution control of the vehicle lamp 200. Specifically, the lamp-side ECU 208 generates an appropriate light distribution pattern based on the information related to the type and position of the object OBJ generated by the calculation processing device 40. The lighting circuit 204 and the optical system 206 work to obtain the light distribution pattern generated by the lamp-side ECU 208.

[0278] In addition, the information on the object OBJ detected by the processing unit 40 can be sent to the vehicle-side ECU 304. The vehicle-side ECU can automatically drive the vehicle based on the information.

[0279] (Implementation 3)

[0280] First, the technical problems related to implementation mode 3 are described.

[0281] The recognition rate of an object recognition system using visible light decreases at night. Therefore, the inventors of the present invention have studied the installation of an active sensor in a car and the use of the obtained image to perform object recognition. The active sensor emits detection light such as infrared light and uses a camera to capture the reflected light from the object.

[0282] The longer the distance light (infrared rays) travels, the more it attenuates. Therefore, the farther away from an object, the less light reaches it, and the less light is reflected from it. That is, even when photographing the same object, the farther away the object is, the lower the pixel value is, thus generating a darker image.

[0283] The classifier used in object recognition uses images that reflect the object to be recognized as training data, performs machine learning, and implements it based on the generated prediction model. Therefore, when images of objects captured at a specific distance are used as training data to generate a prediction model, the recognition rate for images of the same object captured at different distances may be significantly reduced. In order to solve this problem, the same object is captured at various distances as training data, which increases the learning cost.

[0284] The invention related to the third embodiment is proposed in view of the related situation, and one of the exemplary purposes of one aspect is to improve the recognition rate of an object recognition system using an active sensor.

[0285] (Overview of Implementation Method 3)

[0286] One embodiment disclosed in this specification relates to an object recognition system. The object recognition system includes: a strobe camera that divides the depth direction into multiple ranges, changes the time difference between light emission and exposure for each range and takes pictures, and generates multiple slice images corresponding to the multiple ranges; an operation processing device that specifies multiple first correction characteristics corresponding to the multiple ranges, uses the corresponding first correction characteristics to correct each of the multiple slice images, and determines the type of the object based on the corrected multiple slice images.

[0287] In this embodiment, a strobe camera is used as an active sensor. In the strobe camera, the distance of each range is known. Therefore, for each range, the attenuation characteristics of the light transmission path or the diffusion angle of the irradiated light are considered, and the first correction characteristic is determined. By correcting the image using the first correction characteristic, the same image can be obtained regardless of the position of the object, which can improve the recognition rate of the object.

[0288] The first correction characteristic corresponding to each range can be changed according to the measurement environment. Infrared rays have the characteristic of being easily absorbed by water. Therefore, in a strobe camera using infrared rays, the attenuation rate is different in rainy days or dense fog and sunny days, and the attenuation rate is also different depending on the humidity. In addition, dust such as PM2.5 also affects the attenuation rate. Therefore, by changing the first correction characteristic according to the measurement environment, the recognition rate can be further improved.

[0289] The processing device may define a plurality of second correction characteristics corresponding to a plurality of ranges, correct each of the plurality of slice images using the corresponding second correction characteristics, and synthesize the plurality of corrected slice images.

[0290] An image obtained by synthesizing and correcting the plurality of slice images may be displayed on a display.

[0291] Another embodiment of the present invention is a processing device. The processing device is used together with a strobe camera to form an object recognition system. The strobe camera divides the depth direction into multiple ranges, changes the time difference between light emission and exposure for each range, and takes pictures to generate multiple slice images corresponding to the multiple ranges. The processing device includes: a first correction unit that maintains multiple first correction characteristics corresponding to the multiple ranges and corrects each of the multiple slice images using the corresponding first correction characteristics; and a classifier that determines the type of the object based on the corrected multiple slice images.

[0292] The operation processing device may also include: a second correction unit, which determines multiple second correction characteristics corresponding to multiple ranges, and uses the corresponding second correction characteristics to correct each of the multiple slice images; a synthesis unit, which synthesizes the multiple slice images corrected by the second correction unit; and an output unit, which outputs the image generated by the synthesis unit to a display.

[0293] Hereinafter, Embodiments 3-1 to 3-4 will be described with reference to the drawings.

[0294] (Implementation Method 3-1)

[0295] Fig.24 This is a block diagram of an object recognition system 10A according to Embodiment 3-1. This object recognition system 10A is mounted on a vehicle such as an automobile or a motorcycle, and determines the type (category or category) of an object OBJ existing around the vehicle.

[0296] The object recognition system 10A mainly includes a strobe camera 20 and a processing device 40A. The strobe camera 20 includes an illumination device 22, an image sensor 24, and a controller 26. The imaging of the strobe camera 20 is divided into a plurality of N (N≥2) ranges RNG1-RNG N Adjacent measuring ranges may overlap in the depth direction at their boundaries.

[0297] The lighting device 22 radiates the detection light L1 toward the front of the vehicle in synchronization with the light emission timing signal S1 supplied from the controller 26. The detection light L1 is preferably infrared light, but is not limited thereto, and may be visible light having a predetermined wavelength or ultraviolet light.

[0298] The image sensor 24 is configured to be able to perform exposure control in synchronization with the imaging timing signal S2 supplied from the controller 26 and to generate a slice image IMG. The image sensor 24 has sensitivity to the same wavelength as the detection light L1 and captures reflected light (return light) L2 reflected from the object OBJ.

[0299] The controller 26 changes the lighting timing signal S1 and the photographing timing signal S2 for each range RNG to change the time difference between the lighting of the lighting device 22 and the exposure of the image sensor 24. The strobe camera 20 generates the plurality of ranges RNG1-RNG. N Corresponding multiple slice images IMG1-IMG N In the i-th slice image IMG i In the RNG, only the corresponding range is captured i The objects contained in .

[0300] Fig.25 It is a diagram for explaining the operation of the strobe camera 20 . Fig.25 The measurement of the i-th range RNG is shown in iThe lighting device 22 is synchronized with the lighting timing signal S1 and emits light during the lighting period τ1 between time t0 and t1. The top part shows a light diagram with the horizontal axis representing time and the vertical axis representing distance. i The distance from the near side boundary is set to d MINi , set the strobe camera 20 to the range RNG i The distance from the far side boundary is set to d MAXi .

[0301] At a certain moment, the light emitted by the lighting device 22 reaches a distance d MINi , the round trip time T of the reflected light returning to the image sensor 24 MINi for

[0302] T MINi =2×d MINi / c.

[0303] c is the speed of light.

[0304] Similarly, at a certain moment, the light emitted by the lighting device 22 reaches a distance d MAXi , the round trip time T of the reflected light returning to the image sensor 24 MAXi for

[0305] T MAXi =2×d MAXi / c.

[0306] When you need to shoot only the range RNG i When the object OBJ is included in the image, the controller 26 generates a photographing timing signal S2 so that at time t2=t0+T MINi Start exposure at time t3 = t1 + T MAXi End the exposure. This is a one-time exposure action.

[0307] When shooting the i-th range RNG i In this case, the controller 26 may repeat the above-mentioned exposure operation multiple times with a predetermined cycle τ2.

[0308] Fig.26 (a) and (b) are diagrams for explaining images obtained by the strobe camera 20. Fig.26 In the example of (a), the object (pedestrian) OBJ2 exists in the range RNG2, and the object (vehicle) OBJ3 exists in the range RNG3. Fig.26 (b) shows that Fig.26 When the slice image IMG1 is captured, the image sensor is exposed only by the reflected light from the range RNG1, and no object image is captured in the slice image IMG1.

[0309] When capturing slice image IMG2, the image sensor is exposed only by the reflected light from range RNG2, and only object image OBJ2 is captured in slice image IMG2. Similarly, when capturing slice image IMG3, the image sensor is exposed only by the reflected light from range RNG3, and only object image OBJ3 is captured in slice image IMG3. According to such a gated camera 20, objects can be captured separately for each range.

[0310] return Fig.24 The calculation processing device 40A is configured to calculate the value of the plurality of ranges RNG1 to RNG2 obtained by strobing the camera 20. N Corresponding multiple slice images IMG1~IMG N , capable of identifying the type of object. The processing device 40A also includes a classifier 42, which is implemented based on a prediction model generated by machine learning. The algorithm of the classifier 42 is not particularly limited, and may be YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Region-based Convolutional Neural Network), SPPnet (Spatial Pyramid Pooling), FasterR-CNN, DSSD (Deconvolution-SSD), Mask R-CNN, etc., or an algorithm to be developed in the future may be used.

[0311] The operation processing device 40A can be implemented by a combination of a processor (hardware) such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a microcomputer, and a software program executed by the processor (hardware). The operation processing device 40A can also be a combination of multiple processors. That is, the operation processing device 40A can also be composed of hardware only.

[0312] The operation processing device 40 includes a first correction unit 50 provided in front of the classifier 42. The first correction unit 50 is defined to correspond to a plurality of ranges RNG1 to RNG N The corresponding correction characteristics (referred to as a correction characteristic group) can be expressed in the form of a function p'=f(p) that takes the pixel value p before correction as an independent variable (input) and outputs the pixel value after correction.

[0313] The first correction unit 50 uses the corresponding correction characteristics f1(p) to f N (p) Correcting multiple slice images IMG1 to IMG N Each of them generates corrected slice images IMGa1 to IMGa N .

[0314] Corresponding to the i-th range RNG i , specifies the correction characteristic f i (p). In one embodiment, the correction characteristic f i (p) does not depend on the position of the pixel, but only uses the pixel value as an independent variable. At this time, the pixel value p of a certain pixel contained in the image before correction and the pixel value p' of the pixel in the image after correction are as described above.

[0315] p'=f i (p)

[0316] relationship.

[0317] Fig. 27 (a) to (d) are diagrams showing several examples of the correction characteristic f. Fig. 27 In this example, the correction characteristic group f1(p) to f N (p) is a linear function, which can be expressed as follows. i is a correction factor determined for the measuring range, and has a larger value for longer measuring ranges. In addition, p' is clamped at the maximum value.

[0318] p'=f i (p) = α i ×p

[0319] This correction characteristic set corresponds to exposure correction.

[0320] Reference Fig. 27 In this example, the correction characteristic group f1(p) to f N (p) has a so-called S-shaped curve. The function of the S-shaped curve is to increase the contrast of the image.

[0321] When the standardized S-shaped curve is expressed as S(p), the correction characteristic can be determined as

[0322] p'=f i (p) = α i ×S(p).

[0323] Alternatively, the correction characteristic can be determined as

[0324] p'=f i (p) = S(α i ×p).

[0325] Reference Fig. 27 In this example, the correction characteristic group f1(p) to f N (p) has a so-called inverted S-shaped curve. The function of the inverted S-shaped curve is to reduce the contrast of the image. When the standardized inverted S-shaped curve is recorded as INVS(p), the correction characteristic can be determined as

[0326] p'=f i (p) = α i ×INVS(p),

[0327] Or determine

[0328] p'=f i (p) = INVS (α i ×p).

[0329] Reference Fig. 27 In this example, the calibration feature set f1(p) to f N (p) indicates the so-called gamma correction.

[0330] p'=f i (p) = p^γ i

[0331] γ i is the γ correction value of the ith range.

[0332] In addition, the correction characteristics are not limited to Fig. 27 In the cases shown in (a) to (d), it can also be a function of degree 2 or higher, and can be expressed using an exponential function or a trigonometric function. Furthermore, the correction characteristic does not necessarily have to be expressed in the form of a function. The operation processing device 40 can maintain a function f(p) describing the correction characteristic, and the input value p obtains the output p, or it can maintain a lookup table that fixes the relationship between the input p and the output p'.

[0333] In addition, Fig. 27 In (a) to (d), the correction characteristic groups f1(p) to f N (p) is represented by a function of the same form but with different parameters (α or γ), but the present invention is not limited to this, and the function may be in a different form for each range.

[0334] In summary, the correction characteristic group f1(p)~f N (p) The object recognition rate of the subsequent classifier 42 may be high.

[0335] return Fig.24 The first correction unit 50 converts the corrected slice images IMGa1 to IMGa N The corrected slice images IMGa1 to IMGa are supplied to the classifier 42. The classifier 42 classifies the corrected slice images IMGa1 to IMGaN Each, determine each slice image IMGa i The type of objects contained in .

[0336] For example, the output of the classifier 42 (referred to as detection data) OUT i (i=1, 2, ...N) includes the i-th image data IMGa i The size information (position information) of each object image contained in the detection data OUT and its type (type) information. The detection data OUT may also include the information of the bounding box of each object. The type information may also show the possibility (belonging probability) of belonging to each of multiple types, and may also include the identifier of the type with the highest belonging probability.

[0337] The above is the configuration of the object recognition system 10A. Next, its operation will be described.

[0338] Fig.28 (a) and (b) are diagrams for explaining the photographing by the strobe camera 20. Fig.28 (a) is a picture of a scene viewed from one side. Fig.28 (b) shows an image captured by the strobe camera 20. In this example, objects (people) OBJ2, OBJ i ,OBJ N Exists in range RNG2, RNG i 、RNG N middle.

[0339] exist Fig.28 In the image shown in (b), the density of the shadows indicates the magnitude of the pixel values. Even when the same object (person) with the same reflectivity is photographed, the pixel value of the object image of the slice image IMG2 corresponding to the closer range is larger than that of the slice image IMG corresponding to the farther range. N The pixel value of the object image.

[0340] Fig.29 (a) and (b) are diagrams for explaining the correction performed by the first correction unit 50. Fig. 27 The correction characteristics of (a) are used to apply different exposure corrections for each range to generate corrected slice images IMGa2 and IMGa i ,IMGa N Specifically, for the slice image IMG2 of the shorter measurement range, the correction characteristic f2(p) is applied to perform negative exposure correction, and for the slice image IMG of the longer measurement range, the correction characteristic f2(p) is applied to perform negative exposure correction. N , apply the correction characteristic f N (p) Positive exposure correction is performed. As a result, in the corrected slice images IMGa2 and IMGa i ,IMGa NIn the figure, the pixel values ​​of the object image (in the figure, the density of the grid lines) are roughly equal.

[0341] If, before correction, the slice images IMG2 and IMG i 、IMG N When input to the classifier 42, it is possible to classify a certain image (such as IMG i ) is correctly identified as a person, but for other images (such as IMG2, IMG N ) may not be correctly identified as a person. i ,IMGa N The data are input to the classifier 42, which can determine whether the objects contained in each are people.

[0342] The above is the operation of the object recognition system 10A. According to the object recognition system 10A, the recognition rate of the object can be improved.

[0343] (Implementation Method 3-2)

[0344] Fig.30 1 is a block diagram of an object recognition system 10B according to Embodiment 3-2. In Embodiment 3-2, the first correction characteristic of each range is changed according to the measurement environment. Since infrared rays are easily absorbed by water, the first correction characteristic is changed so that the correction degree is enhanced in rainy days or heavy fog.

[0345] For the first correction unit 50 of the operation processing device 40B, a plurality of first correction characteristic groups are specified corresponding to a plurality of environments. The first correction characteristic group can be specified for environments where the transmission characteristics (attenuation characteristics) of the detection light L1 and the reflected light L2 are different. For example, a plurality of first correction characteristic groups can be specified corresponding to the weather (sunny, rainy, cloudy, foggy), or can be specified corresponding to the time period of measurement (day, night, evening). In the case where the object recognition system 10 is mounted on a mobile body such as a car, a plurality of first correction characteristic groups can be specified corresponding to a plurality of driving scenes.

[0346] The object recognition system 10B includes an environment determination unit 70. The environment determination unit 70 determines the current measurement environment (weather, time period, driving scene), and notifies the first correction unit 50. The first correction unit 50 selects a first correction characteristic group from a plurality of first correction characteristic groups based on the notification from the environment determination unit 70. Then, the image of each range is corrected using the selected first correction characteristic group.

[0347] According to implementation 3-2, the recognition rate can be further improved compared with implementation 3-1.

[0348] (Implementation 3-3)

[0349] Fig.31 1 is a diagram showing an object recognition system 10C according to Embodiment 3-3. In the object recognition system 10C, the processing device 40C further includes a second correction unit 52 and a synthesis unit 54 .

[0350] In the second calibration unit 52, a plurality of ranges RNG1 to RNG are defined. N The corresponding plurality of second correction characteristics (referred to as a second correction characteristic group) can be expressed in the form of a function p'=g(p) that takes the pixel value p before correction as an independent variable (input) and outputs the pixel value after correction as similar to the first correction characteristic.

[0351] The second correction unit 52 uses the corresponding second correction characteristics g1(p) to g N (p) Correcting multiple slice images IMG1 to IMG N Each generates corrected slice images IMGb1 to IMGb N .

[0352] Corresponding to the i-th range RNG i , specifies the second correction characteristic g i (p). In one embodiment, the second correction characteristic g i (p) does not depend on the position of the pixel, but only takes the pixel value as an independent variable. In this case, the relationship between the pixel value p of a certain pixel contained in the image before correction and the pixel value p' of the pixel in the image after correction is as described above.

[0353] p'=g i (p)

[0354] relationship.

[0355] Corrected slice images IMGb1~IMGb N The image is input to the synthesis unit 54. The synthesis unit 54 synthesizes the corrected slice images IMGb1 to IMGb2 again. N , generate a slice image IMG c The synthesized slice image IMG c It is displayed on the display 80 and presented to the user.

[0356] The object recognition system 10C is installed in a car, for example. In this case, the display 80 is provided on the driver's seat so that the resynthesized infrared slice image IMGc can be presented to the driver.

[0357] The above is the configuration of the object recognition system 10C. Next, its operation will be described. Since the improvement of the recognition rate in the classifier 42 has been described, the image display on the display 80 will be described.

[0358] Fig.32 (a) to (d) are diagrams for explaining image display of the object recognition system 10C. Fig.32 (a) shows a stereoscopic view of the measurement scene. Object (person) OBJ2 and object (car) OBJ6 exist in each of the ranges RNG2 and RNG6.

[0359] Fig.32 (b) shows Fig.32 In the scene (a), the slice images IMG2 and IMG6 obtained by the gated camera 20. In the slice image IMG2 of the closer range RNG2, the object image of the object OBJ2 has a larger pixel value. On the contrary, in the slice image IMG6 of the farther range RNG6, the object image of the object OBJ6 has a smaller pixel value. Fig.32 In (b) to (d) of FIG. 1 , the pixel value is represented by the density of dots. The position with low density of dots is displayed darker on the display.

[0360] When the general Fig.32 When the two slice images IMG2 and IMG6 in (b) are directly synthesized and displayed on the display, the person OBJ2 on the front side is displayed brightly, while the car OBJ6 on the back side is displayed very darkly, so it is difficult for the user to recognize the car OBJ6.

[0361] In order to solve this problem, in the embodiment 3-3, the slice images IMG2 and IMG6 are corrected by the second correction unit 52. In the corrected slice images IMGb2 and IMGb6, the pixel values ​​of the two object images OBJ2 and OBJ6 are in a close order, and the contrast is optimized in a way that is easy for the user to see as needed. N , generating a synthetic slice image IMGc. In the synthetic slice image IMGc, since the pixel values ​​of the two object images OBJ2 and OBJ6 are in a close order, when displayed on the display 80, the user can easily recognize both the person OBJ2 and the car OBJ6.

[0362] (Implementation 3-4)

[0363] Fig.33 is a diagram showing an object recognition system 10D according to Embodiment 3-4. In Embodiment 3-4, the same range RNG i The first correction characteristic f i and the second correction characteristic g i In this case, the second correction unit 52 is omitted, and the corrected slice images IMGa1 to IMGa N Input synthesis unit 54.

[0364] A variation example related to Implementation Example 3 is described.

[0365] (Variant 3-1)

[0366] In the embodiment, when correcting one image, all pixels are corrected according to the same correction characteristic, but the correction characteristic may also have position dependency. That is, the correction characteristic may be defined as a function with the position of the pixel as an independent variable in addition to the pixel value p.

[0367] (use)

[0368] Fig.34 1 is a block diagram of a car having an object recognition system 10. The car 300 has headlights 302L and 302R. The object recognition system 10 is built into at least one of the headlights 302L and 302R. The headlight 302 is located at the front end of the car body and is the most advantageous location for the strobe camera 20 to detect surrounding objects.

[0369] Fig.35 2 is a block diagram showing a vehicle lamp 200 having an object detection system 210. The vehicle lamp 200 and the vehicle-side ECU 304 together constitute a lamp system 310. The vehicle lamp 200 includes a light source 202, a lighting circuit 204, and an optical system 206. The vehicle lamp 200 is provided with an object detection system 210. The object detection system 210 corresponds to the above-mentioned object recognition system 10, and includes a strobe camera 20 and an operation processing device 40.

[0370] The information related to the object OBJ detected by the processing unit 40 can be used in the light distribution control of the vehicle lamp 200. Specifically, the lamp-side ECU 208 generates an appropriate light distribution pattern based on the information related to the type and position of the object OBJ generated by the processing unit 40. The lighting circuit 204 and the optical system 206 work to obtain the light distribution pattern generated by the lamp-side ECU 208.

[0371] In addition, the information related to the object OBJ detected by the processing unit 40 can be sent to the vehicle-side ECU 304. The vehicle-side ECU can perform automatic driving based on the information.

[0372] (Implementation 4)

[0373] First, the technical issues related to Implementation Method 4 are described.

[0374] When driving at night or in the evening, it is difficult for the driver to see the surrounding objects. In order to compensate for this, the inventor of the present invention has studied the image display system, and the inventor has installed an active sensor using infrared rays on the car, and displayed the obtained image on the display to prompt the driver.

[0375] Fig.36 (a) and (b) are diagrams for explaining the characteristics of a conventional active sensor. Fig.36 (a) is a picture of a shooting scene viewed from the horizontal direction. Fig.36 (b) shows the active sensor shooting Fig.36 (a) is an image of a captured scene.

[0376] The longer the transmission distance of light (infrared rays), the more it attenuates. Therefore, the farther away from the object, the less light reaches the object, and the less light is reflected from the object. In other words, if an active sensor is used, a closer object is reflected brightly, and a farther object is reflected darkly. Therefore, if it is displayed on a display in this state, the way the object is viewed will greatly differ depending on the distance.

[0377] The invention according to the fourth embodiment has been proposed in view of such a situation, and one of the exemplary purposes of one aspect thereof is to provide an image display system capable of displaying an image that is easy to see.

[0378] (Overview of Implementation Method 4)

[0379] One embodiment disclosed in this specification relates to an image display system. The image display system includes: a strobe camera that divides the depth direction into a plurality of ranges, changes the time difference between light emission and exposure for each range and takes pictures to generate a plurality of slice images corresponding to the plurality of ranges; and a processing device that synthesizes the plurality of slice images to generate a synthesized image for display on a display.

[0380] In this embodiment, a gated camera is used as an active sensor. In the gated camera, by adjusting the exposure (shutter speed or sensitivity) and the irradiation intensity of the detection light for each range, it is possible to eliminate the imbalance of exposure depending on the distance to the object. By synthesizing a plurality of slice images obtained in this way, an image that is easy to see can be generated.

[0381] A plurality of correction characteristics corresponding to a plurality of ranges may be defined in the arithmetic processing device, each of the plurality of slice images may be corrected using the corresponding correction characteristics, and the plurality of corrected slice images may be synthesized.

[0382] In a gated camera, objects of each range can be photographed separately, and the distance of each range is known. Therefore, for each range, the attenuation characteristics of the light transmission path or the diffusion angle of the irradiated light are considered, and the correction characteristics are determined. By correcting the image using the correction characteristics, the same image can be obtained regardless of the distance to the object. At this time, the shutter speed or sensitivity of each range can be adjusted.

[0383] The correction characteristics corresponding to each range can be changed according to the measurement environment. For example, infrared rays have the characteristic of being easily absorbed by water. Therefore, in a strobe camera using infrared rays, the attenuation rate is different in rainy days or dense fog and sunny days, and the attenuation rate is also different depending on the humidity. In addition, dust such as PM2.5 also affects the attenuation rate. Therefore, by changing the first correction characteristic according to the measurement environment, the ease of viewing can be further improved.

[0384] Next, embodiments 4-1 to 4-3 will be described with reference to the accompanying drawings.

[0385] (Implementation method 4-1)

[0386] Fig.37 This is a block diagram of an image display system 11E according to Embodiment 4-1. This image display system 11E is mounted on a vehicle such as an automobile or a motorcycle, and presents an image of the surroundings (for example, the front) of the vehicle to the driver.

[0387] The image display system 11E mainly includes a strobe camera 20E, a processing device 40E and a display 80. The strobe camera 20E includes an illumination device 22, an image sensor 24 and a controller 26. The image of the strobe camera 20E is divided into a plurality of N (N≧2) ranges RNG1 to RNG2 in the depth direction. N Adjacent measuring ranges may overlap in the depth direction at their boundaries.

[0388] The lighting device 22 radiates the detection light L1 toward the front of the vehicle in synchronization with the light emission timing signal S1 supplied from the controller 26. The detection light L1 is preferably infrared light, but is not limited thereto, and may be visible light having a predetermined wavelength.

[0389] The image sensor 24 is configured to generate a slice image IMG by performing exposure control in synchronization with an imaging timing signal S2 supplied from the controller 26. The image sensor 24 has sensitivity to the same wavelength as the detection light L1 and captures reflected light (return light) L2 reflected from the object OBJ.

[0390] The controller 26 changes the lighting timing signal S1 and the photographing timing signal S2 for each range RNG to change the time difference between the lighting of the lighting device 22 and the exposure of the image sensor 24. The strobe camera 20E generates the plurality of ranges RNG1-RNG. N Corresponding multiple slice images IMG1-IMG N In the i-th slice image IMG i In the RNG, only the corresponding range is captured i The objects contained in .

[0391] Fig.38It is a diagram for explaining the operation of the strobe camera 20E. Fig.38 The measurement of the i-th range RNG is shown in i The lighting device 22 is synchronized with the lighting timing signal S1 and emits light during the lighting period τ1 between time t0 and t1. The top part shows a light chart with the horizontal axis representing time and the vertical axis representing distance. i The distance from the near side boundary is set to d MINi , set the strobe camera 20 to the range RNG i The distance from the far side boundary is set to d MAXi .

[0392] At a certain moment, the light emitted by the lighting device 22 reaches a distance d MINi , the round trip time T of the reflected light returning to the image sensor 24 MINi for

[0393] T MINi =2×d MINi / c.

[0394] c is the speed of light.

[0395] Similarly, at a certain moment, the light emitted by the lighting device 22 reaches a distance d MAXi , the round trip time T of the reflected light returning to the image sensor 24 MAXi for

[0396] T MAXi =2×d MAXi / c.

[0397] When you need to shoot only the range RNG i When the object OBJ is included in the image, the controller 26 generates a photographing timing signal S2 so that at time t2=t0+T MINi Start exposure at time t3 = t1 + T MAXi End the exposure. This is a one-time exposure action.

[0398] When shooting the i-th range RNG i In this case, the controller 26 may repeat the above-mentioned exposure operation multiple times with a predetermined cycle τ2.

[0399] In this embodiment, in order to ensure that the exposure (brightness value of the object image in the image) has no deviation in each range, the camera 20E is selected to optimize the shutter speed (exposure time), number of exposures, sensitivity, irradiation intensity of the detection light, etc. for each range.

[0400] Fig.39 (a) and (b) are diagrams for explaining images obtained by the strobe camera 20E. Fig.39 In the example of (a), the object (pedestrian) OBJ2 exists in the range RNG2, and the object (vehicle) OBJ3 exists in the range RNG3. Fig.39 (b) shows that Fig.39 When the slice image IMG1 is captured, the image sensor is exposed only by the reflected light from the range RNG1, and no object image is captured in the slice image IMG1.

[0401] When capturing slice image IMG2, the image sensor is exposed only by the reflected light from range RNG2, and only object image OBJ2 is captured in slice image IMG2. Similarly, when capturing slice image IMG3, the image sensor is exposed only by the reflected light from range RNG3, and only object image OBJ3 is captured in slice image IMG3. According to such a gated camera 20E, objects can be captured separately for each range.

[0402] return Fig.37 The processing device 40E synthesizes the multiple ranges RNG1 to RNG obtained by the strobe camera 20E. N Corresponding multiple slice images IMG1~IMG N , generating a synthetic slice image IMGc for display on the display 80.

[0403] The operation processing device 40E can be implemented by a combination of a processor (hardware) such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a microcomputer, a GPU (Graphics Processing Unit), and a software program executed by the processor (hardware). The operation processing device 40E can also be a combination of multiple processors. Alternatively, the operation processing device 40E can also be composed of hardware only.

[0404] The above is the configuration of the image display system 11E. Next, its operation will be described. Fig.40 (a) and (b) are diagrams for explaining the photographing by the strobe camera 20E. Fig.40 (a) is a diagram of a measurement scene viewed from the horizontal direction. Fig.40 (b) shows an image captured by the strobe camera 20E. Fig.40 The scene (a) is similar to Fig.36 In this example, objects (people) OBJ2 and OBJ i ,OBJ N Exists in range RNG2, RNG i 、RNGN middle.

[0405] exist Fig.40 In the image shown in (b), the density of the grid lines indicates the size of the pixel value. In this embodiment, when photographing the same object (person) with the same reflectivity, the pixel value of the object image of the slice image IMG2 corresponding to the closer range is greater than the pixel value of the slice image IMG corresponding to the farther range. N In order to make the pixel values ​​of the object image approximately equal, the camera 20E is selected to perform hardware or software optimization.

[0406] Fig.41 (a) shows the synthesis Fig.40 (b) is a composite image obtained by taking multiple slice images of FIG. Fig.41 In (b), it is shown that Fig.40 The image obtained by the existing infrared active sensor in the scene (a) (i.e., Fig.36 (b) Image). Fig.41 As shown in (b), in the existing infrared active sensor, even when photographing the same object, the pixel value of the farther object is reduced, and the pixel value of the closer object is increased. Therefore, a brighter object or a darker object may exist in one image, and it is difficult to see when it is displayed on the display.

[0407] In contrast, the image display system 11E according to the embodiment 4-1 generates slice images IMG1 to IMG2 having substantially the same exposure (brightness) by strobing the camera 20E. N , so by synthesizing them, it is possible to improve the ease of viewing compared to the past.

[0408] (Implementation method 4-2)

[0409] In the embodiment 4-1, a plurality of slice images IMG1 to IMG N However, such optimization is difficult to perform depending on the type or number of the selected cameras, and the effect may be insufficient. Embodiment 4-2 is particularly useful in such an example.

[0410] Fig.42 4 is a block diagram of an image display system 11F according to Embodiment 4-2. In the image display system 11F, the arithmetic processing device 40F includes a correction unit 52 and a synthesis unit 54 .

[0411] In the calibration unit 52, a plurality of ranges RNG1 to RNG NThe corresponding correction characteristics (referred to as a correction characteristic group) are expressed in the form of a function p'=g(p) that takes the pixel value p before correction as an independent variable (input) and outputs the pixel value after correction.

[0412] The correction unit 52 uses the corresponding correction characteristics g1(p) to g N (p) Correcting multiple slice images IMG1 to IMG N Each generates corrected slice images IMGb1 to IMGb N .

[0413] Corresponding to the i-th range RNG i , specifies the correction characteristic g i (p). In one embodiment, the correction characteristic g i (p) does not depend on the position of the pixel, but only uses the pixel value as an independent variable. At this time, the pixel value p of a certain pixel contained in the image before correction and the pixel value p' of the pixel in the image after correction are as described above.

[0414] p'=gi(p)

[0415] relationship.

[0416] Fig.43 (a) to (d) are diagrams showing several examples of the correction characteristic g. Fig.43 In this example, the correction characteristic group g1(p) to g N (p) is a linear function, which can be expressed as follows. i is a correction factor determined for the measuring range, and has a larger value for longer measuring ranges. In addition, p' is clamped at the maximum value.

[0417] p'=g i (p) = α i ×p

[0418] This correction characteristic set corresponds to exposure correction.

[0419] Reference Fig.43 In this example, the correction characteristic group g1(p) to g N (p) has a so-called S-shaped curve. The function of the S-shaped curve is to increase the contrast of the image.

[0420] When the standardized S-shaped curve is expressed as S(p), the correction characteristic can be determined as

[0421] p'=g i (p) = α i ×S(p).

[0422] Alternatively, the correction characteristic can be determined as

[0423] p'=g i (p) = S(α i ×p).

[0424] Reference Fig.43 In this example, the correction characteristic group g1(p) to g N (p) has a so-called inverted S-shaped curve. The function of the inverted S-shaped curve is to reduce the contrast of the image. When the standardized inverted S-shaped curve is recorded as INVS(p), the correction characteristic can be determined as

[0425] p'=g i (p) = α i ×INVS(p).

[0426] Or determine

[0427] p'=g i (p) = INVS (α i ×p).

[0428] Reference Fig.43 In this example, the correction characteristic group g1(p) to g N (p) indicates the so-called gamma correction.

[0429] p'=g i (p) = p^γ i

[0430] γ i is the γ correction value of the ith range.

[0431] In addition, the correction characteristics are not limited to Fig.43 In the cases shown in (a) to (d), it can also be a function of the second order or higher, and can be expressed using an exponential function or a trigonometric function. Furthermore, the correction characteristic does not necessarily have to be expressed in the form of a function. The operation processing device 40 can maintain a function g(p) that describes the correction characteristic, and the input value p obtains the output p', or it can maintain a lookup table that specifies the relationship between the input p and the output p'.

[0432] In addition, Fig.43 In (a) to (d), the correction characteristic groups g1(p) to g N (p) is represented by a function of the same form but with different parameters (α or γ), but the present invention is not limited to this, and the function may be in a different form for each range.

[0433] In summary, the correction characteristic group g1(p)~g N (p) For each range, the range may be determined so that the same object image is obtained when the same object is photographed.

[0434] return Fig.42 Corrected slice images IMGb1~IMGb N The image is input to the synthesis unit 54. The synthesis unit 54 synthesizes the corrected slice images IMGb1 to IMGb2 again. N , generate a synthetic slice image IMG c The synthesized slice image IMG c It is displayed on the display 80 and presented to the user.

[0435] The above is the configuration of the image display system 11F. Next, its operation will be described. Fig.44 (a) and (b) are explanations Fig.42 FIG. 5 is a diagram of the processing of the correction unit 52. Fig.44 (a) shows a plurality of slice images IMG1 to IMG2 obtained by the gate camera 20. N The farther the range of the image, the smaller the pixel value, and the closer the range of the image, the larger the pixel value.

[0436] Fig.44 (b) shows the slice images IMGb1 to IMGb2 after correction by the correction unit 52. N By synthesizing the corrected slice images IMGb1~IMGb N , we can get Fig.41 (a) is an easy-to-see synthetic slice image IMG c .

[0437] (Implementation method 4-3)

[0438] Fig.45 4-3 is a block diagram of an image display system 11G. In this embodiment, the correction characteristics of each range are changed in accordance with the measurement environment. Since infrared rays are easily absorbed by water, the first correction characteristic is changed so that the degree of correction is enhanced in rainy days or foggy days.

[0439] For the correction unit 52 of the operation processing device 40G, multiple correction characteristic groups are specified corresponding to multiple environments. The correction characteristic group can be specified for environments with different transmission characteristics (attenuation characteristics) of the detection light L1 and the reflected light L2. For example, multiple correction characteristic groups can be specified corresponding to the weather (sunny, rainy, cloudy, foggy), or corresponding to the time period of measurement (day, night, evening). When the multi-image display system 11 is installed on a mobile body such as a car, multiple correction characteristic groups can be specified corresponding to multiple driving scenes.

[0440] The image display system 11G includes an environment determination unit 70. The environment determination unit 70 determines the current measurement environment (weather, time period, driving scene), and notifies the correction unit 52. The correction unit 52 selects a correction characteristic group from a plurality of correction characteristic groups based on the notification from the environment determination unit 70. Then, the image of each range is corrected using the selected correction characteristic group.

[0441] According to implementation 4-3, the recognition rate can be further improved compared with implementation 4-2.

[0442] A variation example related to Implementation Example 4 is described.

[0443] (Variant 4-1)

[0444] In the embodiment, when correcting one image, all pixels are corrected according to the same correction characteristic, but the correction characteristic may also have position dependency. That is, the correction characteristic may be defined as a function with the position of the pixel as an independent variable in addition to the pixel value p.

[0445] (Implementation method 5)

[0446] First, the issues related to the fifth embodiment will be described.

[0447] When verifying the operation of the strobe camera or the system in which it is installed, it is necessary to select several or all ranges as the inspection target ranges, configure reference reflectors for the inspection target ranges, and shoot with the strobe camera to determine whether the shooting is normal. When the strobe camera is used for vehicle-mounted purposes, it is necessary to shoot objects with a range of tens of meters to more than 100 meters. Therefore, when a longer range is selected as the inspection target range, there is a problem that a larger space is required during the inspection.

[0448] Furthermore, even if a large space can be secured, when a plurality of inspection target ranges are selected, it is necessary to perform inspection while changing the position of the reference reflector, so the number of operations required for inspection increases, which causes cost increase.

[0449] The invention related to embodiment 5 is obtained in view of such a situation, and one of the exemplary purposes of one scheme is to provide a gating camera or its inspection method that can perform inspection using a reference reflector configured in other ranges while focusing on a certain range.

[0450] (Overview of Implementation Method 5)

[0451] One embodiment disclosed in this specification relates to a gated camera that divides the depth direction into multiple ranges and generates multiple slice images corresponding to the multiple ranges. The gated camera includes: an illumination device that irradiates detection light according to a light emission timing signal; an image sensor that captures reflected light according to a photography timing signal; and a controller that generates a light emission timing signal and a photography timing signal with a time difference specified for each range. The camera is configured to give an offset to the time difference between the light emission timing signal and the photography timing signal during inspection.

[0452] When a certain range is set as the photographic object, if the controller is operated, a light emitting timing signal and a photographing timing signal having a time difference corresponding to the range are generated. When a negative offset is given to the time difference between the light emitting timing signal and the photographing timing signal, an object existing in a range closer than the set range is photographed, and when a positive offset is given to the time difference between the light emitting timing signal and the photographing timing signal, an object existing in a range farther than the set range is photographed. Thus, when a certain range is set as the photographic object, a reference reflector configured in another range can be used for inspection.

[0453] Alternatively, the strobe camera can be connected to an external delay unit on the path of the light-emitting timing signal. Alternatively, the strobe camera can also include a delay unit that can be inserted into the path of the light-emitting timing signal during inspection. Thus, in a state where a relatively long range is set as the photographic object, a reference reflector configured in a relatively short range can be used for inspection. In addition, the space required for inspection can be reduced.

[0454] Alternatively, the strobe camera can be connected to an external delay unit on the path of the photographic timing signal. Alternatively, the strobe camera can further include a delay unit that can be inserted into the path of the photographic timing signal during inspection. Thus, in a state where a relatively close range is set as the photographic object, a reference reflector arranged at a relatively far range can be used for inspection.

[0455] In one embodiment, the strobe camera includes: an illumination device for irradiating detection light according to a light emission timing signal; an image sensor for capturing reflected light according to a photographic timing signal; and a controller for generating a light emission timing signal and a photographic timing signal with a time difference specified for each range. During inspection, the light emission timing signal can be output to an external reference light source via an external delay unit. Thus, the strobe camera can be inspected using an external reference light source. In addition, the space required for inspection can be reduced.

[0456] In one embodiment, the strobe camera includes: an illumination device for irradiating detection light according to a light emission timing signal; an image sensor for capturing reflected light according to a photographic timing signal; and a controller for generating a light emission timing signal and a photographic timing signal with a time difference specified for each range. During inspection, the photographic timing signal can be output to an external image sensor via an external delay unit. Thus, the strobe camera can be inspected using an external image sensor. In addition, the space required for inspection can be reduced.

[0457] Next, embodiment 5 will be described with reference to the drawings.

[0458] (Implementation method 5-1)

[0459] Fig.46 2 is a block diagram of the gate camera 20 according to the embodiment 5-1. The gate camera 20 divides the depth direction into a plurality of N (N≧2) ranges RNG1 to RNG N Take pictures and generate multiple ranges RNG1~RNG N Corresponding multiple images (slice images) IMG1~IMG N Adjacent measuring ranges may also overlap each other at their boundaries in the depth direction.

[0460] The strobe camera 20 includes an illumination device 22 , an image sensor 24 , and a controller 26 .

[0461] The lighting device 22 irradiates the detection light L1 toward the front of the vehicle in synchronization with the light emission timing signal S1 supplied from the controller 26. The detection light L1 is preferably infrared light, but is not limited thereto, and may be visible light or ultraviolet light having a predetermined wavelength.

[0462] The image sensor 24 is configured to be able to perform exposure control in synchronization with the imaging timing signal S2 supplied from the controller 26 and to generate a slice image IMG. The image sensor 24 has sensitivity to the same wavelength as the detection light L1 and captures reflected light (return light) L2 reflected from the object OBJ.

[0463] In the controller 26, the pulse widths of the light emission timing signal S1 and the photographing timing signal S2 and their time difference are defined for each range. The light emission timing signal S1 and the photographing timing signal S2 defined for the i-th (1≦i≦N) range are represented by S1. i 、S2 i The range number i is added to mark it. i As the object of photography, it is called focusing on the i-th range RNG i This is consistent with the selection of the light-emitting timing signal S1 i , Photography timing signal S2 iIn addition, the RNG i It is called the focal range. In addition, it should be noted that this focus is different from the focus in the optical sense.

[0464] Fig.47 It is a diagram for explaining the basic operation of the strobe camera 20. Fig.47 The measurement of the i-th range RNG is shown in i The lighting device 22 and the light-emitting timing signal S1 i Synchronize, light is emitted between the light emitting devices τ1 between time t0 and t1. The top part shows a light diagram with the horizontal axis representing time and the vertical axis representing distance. i The distance of the closer boundary is d MINi will be from the gate camera 20 to the range RNG i The distance from the inner boundary is d MAXi .

[0465] At a certain moment, the light emitted by the lighting device 22 reaches a distance d MINi , the round trip time T of the reflected light returning to the image sensor 24 MINi for

[0466] T MINi =2×d MINi / c.

[0467] c is the speed of light.

[0468] Similarly, at a certain moment, the light emitted by the lighting device 22 reaches a distance d MAXi , the round trip time T of the reflected light returning to the image sensor 24 MAXi for

[0469] T MAXi =2×d MAXi / c.

[0470] When you need to shoot only the range RNG i When the object OBJ is included in the image, the controller 26 generates a photographing timing signal S2 so that at time t2=t0+T MINi Start exposure at time t3 = t1 + T MAXi End the exposure. This is a one-time exposure action.

[0471] When shooting the i-th range RNG i In this case, the controller 26 may repeat the above-mentioned exposure operation multiple times with a predetermined cycle τ2.

[0472] Fig.48 (a) and (b) are diagrams for explaining images obtained by the strobe camera 20. Fig.48 In the example of (a), the object (pedestrian) OBJ2 exists in the range RNG2, and the object (vehicle) OBJ3 exists in the range RNG3. Fig.48 (b) shows that Fig.48 When the slice image IMG1 is captured, the image sensor is exposed only by the reflected light from the range RNG1, and no object image is captured in the slice image IMG1.

[0473] When capturing slice image IMG2, the image sensor is exposed only by the reflected light from range RNG2, and only object image OBJ2 is captured in slice image IMG2. Similarly, when capturing slice image IMG3, the image sensor is exposed only by the reflected light from range RNG3, and only object image OBJ3 is captured in slice image IMG3. According to such a gated camera 20, objects can be captured separately for each range.

[0474] By strobing the camera 20, multiple ranges RNG1 to RNG are generated. N Corresponding multiple slice images IMG1~IMG N In the i-th slice image IMG i In the RNG, only the corresponding range is reflected i The objects contained in .

[0475] return Fig.46 The strobe camera 20 is configured to provide an offset τ to the time difference Δt between the light emission timing signal S1 and the photographing timing signal S2 during the inspection process. In the present embodiment, the strobe camera 20 can introduce a negative offset τ. The negative offset -τ can be obtained by making the previous light emission timing signal S1 j Delay to import.

[0476] In order to introduce the negative offset -τ, a delay unit 28 is provided externally to the strobe camera 20, or the delay unit 28 is built in. During normal shooting, the delay unit 28 does not affect the shooting, and the light-emission timing signal S1 generated by the controller 26 is directly supplied to the lighting device 22 (path i). In the inspection process, the light-emission timing signal S1 generated by the controller 26 passes through the delay unit 28, and is delayed by a time τ corresponding to the offset, and the delayed light-emission timing signal S1' is supplied to the lighting device 22 (path ii).

[0477] The above is the configuration of the gate camera 20. Next, the inspection by the gate camera 20 will be described.

[0478] Fig.49: is a diagram showing the layout of the strobe camera 20 and the reference reflector REF in the inspection process. In this example, it is checked whether the shooting of the j-th range RNGj is normal. The reference reflector REF is arranged at the i-th range RNGi which is closer to the j-th range RNGj. The controller 26 of the strobe camera 20 is set to a state in which the j-th range RNGj can be photographed, that is, the focus is focused on the j-th range RNGj, that is, the controller 26 outputs the light-emitting timing signal S1j and the photographing timing signal S2j. In this case, a negative offset -τ is introduced into the time difference Δtj between the light-emitting timing signal S1j and the photographing timing signal S2j. The offset τ is τ = (dMINj-dMINi) / c×2.

[0479] Fig.50 Yes Description Fig.46 FIG. 20 is a diagram of an inspection of the strobe camera 20. Light emission timing signal S1 j The time difference with the photographing timing signal S2j is Δt j . By emitting the timing signal S1 j The delayed light emitting timing signal S1 is delayed by τ by the delay unit 28. j’ With photography timing signal S2 j The time difference is Δt j -τ.

[0480] exist Fig.50 In the upper graph, the single-dot chain line shows the j The light when the lighting device 22 is illuminated. This light is in the jth range RNG j Reflection, according to the photography timing signal S2 j During the inspection process, the lighting device 22 is not based on the light emission timing signal S1 j , but according to the delayed S1 j 'Emit light. The solid line shows the light at this time. This light is measured by the range RNG i The object (reference reflector REF) reflects, according to the photography timing signal S2 j Photograph reflected light.

[0481] As a result, the captured slice image IMG j If the reference reflector REF is reflected, the strobe camera 20 is judged to be normal, and if it is not reflected, it is judged to be abnormal. The above is the inspection of the strobe camera 20.

[0482] According to the strobe camera 20, by the light-emitting timing signal S1 j And photography timing signal S2 j The time difference Δt j By supplying a negative offset -τ, it is possible to capture the RNG that is larger than the set range. jCloser range RNG i The reference object REF in .

[0483] Fig.51 It is shown Fig.46 FIG. 1 is a diagram of a first configuration example of a strobe camera 20. The strobe camera 20 has a built-in delay unit 28. For example, the strobe camera 20 includes selectors 30 and 32 disposed in front of and behind the delay unit 28. The selectors 30 and 32 bypass the delay unit 28 during normal photographing (non-inspection process) and directly supply the light emission timing signal S1 to the lighting device 22. In addition, during the inspection process, the selectors 30 and 32 supply the delayed light emission timing signal S1' to the lighting device 22 via the delay unit 28 via the light emission timing signal S1.

[0484] Fig.52 It is shown Fig.46 FIG. 2 is a diagram of a second configuration example of the selection camera 20. Fig.52 The strobe camera 20 and Fig.51 The strobe camera 20 is different in that the delay unit 28 is detachable.

[0485] (Implementation method 5-2)

[0486] Fig.53 This is a block diagram of a strobe camera 20A in Embodiment 5-2. In Embodiment 5-2, in the inspection process, the strobe camera 20A can give a positive offset to the time difference Δt between the light emission timing signal S1 and the photographing timing signal S2. The positive offset can be introduced by delaying the subsequent photographing timing signal S2 by a time τ.

[0487] In order to introduce a positive offset τ, a delay unit 28 is provided externally to the strobe camera 20A, or a delay unit 28 is built in. During normal photography, the delay unit 28 does not affect photography, and the photography timing signal S2 generated by the controller 26 is supplied to the image sensor 24 as it is (path i). During the inspection process, the photography timing signal S2 generated by the controller 26 is delayed by a time τ corresponding to the offset through the delay unit 34, and the delayed photography timing signal S2' is supplied to the image sensor 24 (path ii).

[0488] Next, for Fig.53 Inspection of the strobe camera 20A will be described. Fig.54 20A and the reference reflector REF in the inspection process. i The reference reflector REF is placed at a position lower than the i-th range RNG. i The jth range RNG on the closer side jThe controller 26 of the strobe camera 20A is set to be able to capture the i-th range RNG i The state of the controller 26 outputs the light-emitting timing signal S1 i , Photography timing signal S2 i In this case, a positive bias +τ is introduced into the light emission timing signal S1 i The time difference Δt from the photographing timing signal S2i i The offset τ is τ=(d MINj -d MINi ) / c×2.

[0489] Fig.55 Yes Description Fig.53 FIG. 20A of the strobe camera for inspection. Light emission timing signal S1 i And photography timing signal S2 i The time difference is Δt i The photographing timing signal S2i is delayed by τ through the delay unit 28, so that the light emission timing signal S1 i The time difference with the delayed imaging timing signal S2i' is Δt i +τ.

[0490] In the inspection process, it is not based on the photographing timing signal S2 i , but according to the delayed S2 i 'Shoot. As a result, if the slice image IMG i If the reference reflector REF is captured in the image, the strobe camera 20A is judged to be normal, and if it is not captured, it is judged to be abnormal.

[0491] according to Fig.53 The gated camera 20A can capture an image of a reference object REF existing at a range RNGj farther than the set range RNGi by supplying a positive offset +τ to the time difference Δti between the light emission timing signal S1i and the photographing timing signal S2i.

[0492] Fig.56 It is shown Fig.53 FIG. 1 is a diagram of a first configuration example of a strobe camera 20A. The strobe camera 20A has a built-in delay unit 34. For example, the strobe camera 20A includes selectors 36 and 38 disposed at the front and rear of the delay unit 34. The selectors 36 and 38 bypass the delay unit 34 during normal photography (non-inspection process) and directly supply the photography timing signal S2 to the image sensor 24. In addition, during the inspection process, the selectors 36 and 38 allow the photography timing signal S2 to pass through the delay unit 34 and supply the delayed photography timing signal S2' to the image sensor 24.

[0493] Fig.57 It is shown Fig.53 FIG. 2 is a diagram showing a second configuration example of the selection camera 20A. Fig.57 The strobe camera 20A and Fig.56 The strobe camera 20A is different in that the delay unit 34 is detachable.

[0494] (Implementation method 5-3)

[0495] Fig.58 : is a diagram showing a gating camera 20B of embodiment 5-3. The gating camera 20B includes a lighting device 22, an image sensor 24, and a controller 26. The gating camera 20B is configured to output a light emission timing signal S1 to an external reference light source 62 via an external delay unit 60 during an inspection process. In embodiment 5-3, the lighting device 22 is not used during the inspection process, and the image sensor 24 and the controller 26 are inspected to see whether they are operating normally. The reference light source 62 is arranged in opposition to the gating camera 20B. The reference light source 62 emits light according to the delayed light emission timing signal S1'. The above is the structure of the gating camera 20B.

[0496] The operation of the strobe camera 20B is substantially the same as Fig.46 The difference is that an external reference light source 62 is used instead of the reference reflector REF. In addition, the lighting device 22 is in a non-lighting state during the inspection process. The reference light source 62 is set to the i-th range RNG i The focus of the strobe camera 20B is aligned with the jth range RNG j .

[0497] exist Fig.58 The optical path length of the emitted light L3 of the reference light source 62 is 1 / 2 of the optical path length of the probe light L1 and the reflected light L2 emitted and reflected by the illumination device 22. Therefore, the delay amount τ of the delay unit 60 is τ=(d MINj -d MINi ) / c.

[0498] The result of shooting, if in the slice image IMG j If the emitted light of the reference light source 62 is reflected in the strobe camera 20B, it is determined that the strobe camera 20B is normal.

[0499] In addition, use Fig.52 The strobe camera 20 can be constructed with Fig.58 The same inspection system of the gated camera 20B.

[0500] (Implementation method 5-4)

[0501] Fig.59: is a diagram showing a gating camera 20C of embodiment 5-4. The gating camera 20C includes an illumination device 22, an image sensor 24, and a controller 26. The gating camera 20C is configured to output a photographing timing signal S2 to an external reference camera 66 via an external delay unit 64 during an inspection process. In embodiment 5-4, the image sensor 24 is not used during the inspection process, and it is inspected whether the illumination device 22 and the controller 26 are operating normally. The reference camera 66 is provided in opposition to the gating camera 20C. The reference camera 66 captures the emitted light L4 of the illumination device 22 based on the delayed photographing timing signal S2'. The above is the structure of the gating camera 20C.

[0502] The inspection by the gate camera 20C is described. The image sensor 24 is not used in the inspection process. The reference camera 66 is set at the jth range RNG j . In the case of being able to measure the i-th range RNG i The strobe camera 20C is set in the state.

[0503] exist Fig.59 The optical path length of the outgoing light L4 of the lighting device 22 is Fig.46 The optical path length of the probe light L1 and the reflected light L2 emitted and reflected by the illumination device 22 is 1 / 2 of that of the reflected light L2. Therefore, the delay amount τ of the delay unit 60 is τ=(d MINj -d MINi ) / c.

[0504] As a result of the photographing, if the emitted light of the lighting device 22 is reflected in the output image of the reference camera 66, the strobe camera 20C is determined to be normal.

[0505] In addition, use Fig.57 The strobe camera 20A can be constructed with Fig.59 The same inspection system of the gated camera 20C.

[0506] (use)

[0507] Fig.60 2 is a block diagram showing a vehicle lamp 200 having an object detection system 210. The vehicle lamp 200 and the vehicle-side ECU 304 together constitute a lamp system 310. The vehicle lamp 200 includes a light source 202, a lighting circuit 204, and an optical system 206. The vehicle lamp 200 is provided with the object detection system 210. The object detection system 210 includes the above-mentioned strobe camera 20 and the operation processing device 40.

[0508] The calculation processing device 40 is configured to obtain the measurement data obtained by the strobe camera 20 and the multiple ranges RNG1 to RNG N Corresponding multiple slice images IMG1~IMGN , capable of identifying the type of object. The processing device 40 includes a classifier 42, which is implemented based on a prediction model generated by machine learning. The algorithm of the classifier 42 is not particularly limited, but YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Region-based Convolutional Neural Network), SPPnet (Spatial Pyramid Pooling), Faster R-CNN, DSSD (Deconvolution-SSD), Mask R-CNN, etc. can be used, or an algorithm to be developed in the future can be used.

[0509] The calculation processing device 40 may be implemented by a combination of a processor (hardware) such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit) or a microcomputer and a software program executed by the processor (hardware). Alternatively, the calculation processing device 40 may be composed only of hardware.

[0510] The information related to the object OBJ detected by the calculation processing device 40 can be used in the light distribution control of the vehicle lamp 200. Specifically, the lamp-side ECU 208 generates an appropriate light distribution pattern based on the information related to the type and position of the object OBJ generated by the calculation processing device 40. The lighting circuit 204 and the optical system 206 work to obtain the light distribution pattern generated by the lamp-side ECU 208.

[0511] In addition, the information related to the object OBJ detected by the processing unit 40 can be sent to the vehicle-side ECU 304. The vehicle-side ECU can perform automatic driving based on the information.

[0512] (Implementation method 6)

[0513] First, a description will be given of technical problems associated with Embodiment 6. The inventors of the present invention studied how to combine a plurality of slice images captured by a strobe camera into one image, and ultimately recognized the following technical problems.

[0514] Fig.61 (a) and (b) are diagrams for explaining problems related to image synthesis. Fig.61 (a) shows a plurality of slice images IMG1 to IMG NFor ease of understanding, consider a one-dimensional image, with pixel positions plotted on the horizontal axis and pixel values ​​plotted on the vertical axis. Here, it is assumed that the object only exists in the i-th range, so in the i-th slice image IMG i The image of the object is reflected. When capturing a slice image IMG of a certain range, if light other than the detection light (external light) enters the infrared camera, it is recorded as noise. The noise component is included in all slice images. Fig.61 (b) shows an image obtained by simply adding and synthesizing a plurality of slice images. When the slice images are simply synthesized, the S / N ratio is greatly reduced.

[0515] In addition, this problem is not generally recognized by those skilled in the art, and further, it is recognized independently by the inventors of the present invention.

[0516] The invention related to the sixth embodiment is proposed in view of the related situation, and one of the exemplary purposes of one aspect is to provide an imaging device capable of generating a composite image with improved image quality.

[0517] (Overview of Implementation Method 6)

[0518] One embodiment disclosed in this specification relates to an imaging device. The imaging device includes: a strobe camera that divides the depth direction into a plurality of ranges, changes the time difference between light emission and exposure for each range and shoots, and generates a plurality of slice images corresponding to the plurality of ranges; and an image processing device that synthesizes the plurality of slice images to generate a synthesized image. The image processing device detects a non-existent area where an object does not exist for each of the plurality of slice images, performs noise reduction processing on the non-existent area, and synthesizes the plurality of slice images after noise reduction.

[0519] After removing the noise in the non-existent region, the image quality of the synthesized image can be improved by synthesizing a plurality of slice images.

[0520] When the image processing device detects an existing area where an object exists in a certain slice image, the area of ​​other slice images overlapping with it is determined as an absent area. The principle of the gated camera is to detect only the reflected light from the object closest to the depth direction. Therefore, in the case where the range does not overlap in the depth direction, the reflection from the object can be detected in a certain area of ​​a certain slice image, and the image detected by other slice images in this area can be regarded as noise and removed.

[0521] The image processing device may set the pixel values ​​of a plurality of pixels included in the non-existent area to zero.

[0522] When there are a plurality of pixels whose pixel values ​​are greater than a threshold value, the image processing device may determine that a region including these pixels is a region where an object exists.

[0523] Next, embodiment 6 will be described with reference to the drawings.

[0524] (Implementation method 6-1)

[0525] Fig.62 1 is a diagram showing an imaging device 10 according to Embodiment 6-1. The imaging device 10 includes a strobe camera 20 and an image processing device 30. The imaging device 10 is mounted in a vehicle such as a car or a motorcycle, for example, and captures an image of an object OBJ existing around the vehicle.

[0526] The camera device 10 mainly includes a strobe camera 20 and an image processing device 30. The strobe camera 20 includes an illumination device 22, an image sensor 24, and a controller 26. The imaging of the strobe camera 20 can be performed by dividing the depth direction into a plurality of N (N≧2) ranges RNG1 to RNGN. Adjacent ranges can overlap each other in the depth direction at their boundaries.

[0527] The lighting device 22 irradiates the detection light L1 toward the front of the vehicle in synchronization with the light emission timing signal S1 supplied from the controller 26. The detection light L1 is preferably infrared light, but is not limited thereto, and may be visible light having a predetermined wavelength or ultraviolet light.

[0528] The image sensor 24 is configured to perform exposure control in synchronization with the imaging timing signal S2 supplied from the controller 26 and output a slice image IMG. The image sensor 24 has sensitivity to the same wavelength as the detection light L1 and captures reflected light (return light) L2 reflected from the object OBJ.

[0529] The controller 26 changes the light emission timing signal S1 and the imaging timing signal S2 for each range RNG, thereby changing the time difference between the light projection of the illumination device 22 and the exposure of the image sensor 24. The strobe camera 20 generates the plurality of ranges RNG1 to RNG2. N Corresponding multiple slice images IMG1~IMG N As a principle, for the i-th slice image IMG i , only reflects the corresponding range RNG i The objects contained in .

[0530] Fig.63 It is a diagram for explaining the operation of the strobe camera 20 . Fig.63 The measurement of the i-th range RNG is shown in i The lighting device 22 is synchronized with the light-emitting timing signal S1 and emits light during the light-emitting period τ1 between time t0 and t1. The light diagram is shown at the top with the horizontal axis representing time and the vertical axis representing distance. i The distance from the near side boundary is set to dMINi , will reach the range RNG i The distance from the far side boundary is set to d MAXi .

[0531] At a certain moment, the light emitted by the lighting device 22 reaches a distance d MINi , the round trip time T until the reflected light returns to the first image sensor 24 MINi for

[0532] T MINi =2×d MINi / c.

[0533] c is the speed of light.

[0534] Similarly, at a certain moment, the light emitted by the lighting device 22 reaches the distance d MAXi , the round trip time T for the reflected light to return to the image sensor 24 MAXi for

[0535] T MAXi =2×d MAXi / c.

[0536] When you need to shoot only the range RNG i When the object OBJ contained in MINi Start exposure at time t3 = t1 + T MAXi The exposure is ended to generate a photographing timing signal S2A. This is one exposure operation.

[0537] When shooting the i-th range RNG i At this time, the controller 26 can repeat the above exposure action multiple times in a predetermined period τ2.

[0538] Fig.64 (a) and (b) are diagrams for explaining images obtained by the strobe camera 20. Fig.64 In the example of (a), the object (pedestrian) OBJ2 exists in the range RNG2, and the object (vehicle) OBJ3 exists in the range RNG3. Fig.64 In (b), it is shown that Fig.64 When capturing image IMG1, the image sensor is exposed only by the reflected light from range RNG1, so that no object image is reflected in image IMG1.

[0539] When capturing image IMG2, the image sensor is exposed only by the reflected light from range RNG2, so only object image OBJ2 is reflected in image IMG2. Similarly, when capturing image IMG3, the image sensor is exposed only by the reflected light from range RNG3, so only object image OBJ3 is reflected in image IMG3. According to such a gated camera 20, objects can be captured separately for each range.

[0540] return Fig.61 The image processing device 30 synthesizes the image obtained by the gate camera 20 with the multiple ranges RNG1 to RNG N Corresponding multiple slice images IMG1~IMG N , generate a synthetic image IMGc.

[0541] The image processing device 30 processes the plurality of slice images IMG1 to IMG N Each of them determines the area where the object does not exist (non-existent area). For example, when there are multiple pixels with pixel values ​​greater than a predetermined threshold value in the slice image IMG#, the image processing device 30 determines the area containing these pixels as the existing area, and determines the area outside of them as the non-existent area. Alternatively, the image processing device 30 performs edge extraction on the slice image IMG#, and determines the area surrounded by the edge as the existing area, and the area outside of it as the non-existent area.

[0542] The image processing device 30 processes the plurality of slice images IMG1 to IMG N In each case, noise reduction treatment is performed on the non-existent area.

[0543] For example, noise removal in the slice image IMG j In the example, the pixel value of the non-existent area can be set to zero. Alternatively, the image processing device 30 can multiply the pixel value of the non-existent area by a coefficient less than 1. The slice image after noise reduction is set as IMGa j .

[0544] Then, the image processing device 30 synthesizes the plurality of slice images IMGa1 to IMGa1 after noise reduction. N , generate a synthetic image IMGc.

[0545] Fig.65 3 is a functional block diagram of the image processing device 30. The image processing device 30 includes an object detection unit 32, a noise reduction unit 34, and a synthesis unit 36. The object detection unit 32 detects a plurality of slice images IMG1 to IMG N The object detection unit 32 supplies information indicating the non-existent area of ​​each slice image to the noise reduction unit 34. The noise reduction unit 34 detects the presence area and the non-existent area of ​​each slice image. NEach of the slice images IMGa1 to IMGa1 after the noise reduction processing is performed on the non-existent region. N , generate a synthetic image.

[0546] The image processing device 30 may also be composed of hardware such as FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit). Alternatively, the image processing device 30 may be implemented by a combination of a processor (hardware) such as a CPU (Central Processing Unit), an MPU (MicroProcessing Unit), or a microcomputer and a software program executed by the processor (hardware). The image processing device 30 may also be a combination of multiple processors.

[0547] The above is the configuration of the imaging device 10. Next, the operation of the imaging device 10 will be described. Fig.66 It is a stereogram of the shooting scene. Here, the number of ranges N is set to 6. Object (person) OBJ2 and object (car) OBJ6 exist in each of the ranges RNG2 and RNG6.

[0548] Fig.67 (a) to (c) are diagrams for explaining the synthesis process performed by the image processing device 30. Fig.67 (a) shows that Fig.66 . N1 represents noise caused by external light. N2 represents noise caused by an object included in an adjacent range.

[0549] Focusing on the slice image IMG2, the object image of the object OBJ2 is clearly captured, and the range including it is determined as the existence area A1, and the outside thereof is determined as the non-existence area. Similarly, focusing on the slice image IMG6, the object image of the object OBJ6 is clearly captured, and the range including it is determined as the existence area A2, and the outside thereof is determined as the non-existence area.

[0550] When focusing on the slice images IMG3 and IMG4, there is no region in which the pixel value exceeds the threshold value, and no contour is detected, so it is determined as a whole that there is no region.

[0551] When focusing on the slice image IMG1, the noise N2 caused by the object OBJ2 of the adjacent range is reflected. In this example, the pixel value of the noise N2 is sufficiently small, and the contour cannot be detected, and the noise N2 is determined to be a non-existent area. The same is true for the slice image IMG5, and the noise N2 is determined to be a non-existent area.

[0552] Fig.67 (b) shows the slice images IMGa1 to IMGa6 to which the noise reduction process has been performed. a In the non-existent area, the pixel value is zero, and the noise components N1 and N2 are removed.

[0553] Fig.67 (c) shows a composite image IMGc obtained by synthesizing the slice images IMGa1 to IMGa6 after noise reduction. Fig.67 As shown in (c), according to the imaging device 10, the noise N1 caused by external light can be removed. In addition, the noise N2 caused by objects in adjacent ranges can also be removed.

[0554] Reference Fig.68 (a) to (c) illustrate the problems that arise in the processing of Implementation 6-1. Fig.68 (a) to (c) are diagrams showing other examples of the operation of the imaging device 10 according to Embodiment 6-1.

[0555] Fig.68 (a) and Fig.67 (a) Similarly, it is shown in Fig.66 Slice images IMG1 to IMG6 are obtained from the scene. Focusing on the slice image IMG1, the noise N2 caused by the adjacent range object OBJ2 is Fig.67 (a) more strongly reflects that the pixel value of noise N2 is large to the extent that the contour can be detected. At this time, the range including noise N2 is mistakenly determined as the existence area A3. Similarly, about the slice image IMG5, the range including noise N2 is mistakenly determined as the existence area A4.

[0556] As a result, Fig.68 In (b), noise N2 remains in the slice images IMGa1 and IMGa5 after noise reduction processing. Fig.68 In the composite image shown in (c), the outline of the object is blurred and the contrast is reduced. To solve this problem, it is necessary to carefully determine the threshold when detecting the existing area. If the threshold is too large, the actual object may be mistaken for noise.

[0557] (Implementation method 6-2)

[0558] In the embodiment 6-2, the problem of the embodiment 6-1 is solved. The overall structure of the camera device 10 is similar to Fig.62 Similarly, the processing of the image processing device 30 is different between Embodiment 6-1 and Embodiment 6-2.

[0559] In Embodiment 6-2, when an existing region exists in a certain slice image IMGi, the image processing device 30 sets a region overlapping with the existing region as an absent region in another slice image IMGj (j≠i).

[0560] Fig.69 (a) to (c) are diagrams illustrating the synthesis processing of embodiment 6-2. Fig.69 (a) and Fig.68 (a) is the same. Fig.69 In the slice image IMG1 of (a), a contour with strong noise N2 is detected. On the other hand, in the other slice image IMG2, the existing area A1 is detected, so that in the slice image IMG1, the area containing noise N2 overlapping with the existing area A1 can be correctly determined as the non-existent area. For the same reason, the area containing noise N2 can be correctly determined as the non-existent area in the slice image IMG5.

[0561] Results, such as Fig.69 As shown in (b), the influence of noise N2 is removed from the noise-corrected slice images IMG1 and IMG5, as shown in FIG. Fig.69 As shown in (c), a high-quality composite image IMGc can be obtained.

[0562] In Embodiment 6-1, slice images are processed independently to determine the presence and absence of areas. In Embodiment 6-2, multiple slice images are referenced to determine the presence and absence of areas based on their relationship. This can prevent erroneous judgments. In particular, the strobe camera has the characteristic of only detecting reflected light from the nearest object, so that an object is not captured simultaneously in the same pixel or the same area of ​​two (or three or more) slice images. In Embodiment 6-2, by utilizing this characteristic, noise and objects can be reliably distinguished.

[0563] Furthermore, when existing regions are detected simultaneously in the overlapping region of two slice images, the existing regions including pixels with larger pixel values ​​in the two slice images may be set as valid, and the other existing regions may be set as non-existent regions.

[0564] (Implementation method 6-3)

[0565] In Embodiments 6-1 and 6-2, the presence area and the absence area are judged and noise reduction processing is performed, but the present invention is not limited to this and processing may be performed for each pixel.

[0566] That is, compare all slice images IMG1 to IMG NThe pixel values ​​of the pixels at the same position of the image are calculated, and the largest one is taken as a valid pixel. The pixels other than that are judged as noise, and the pixel values ​​are set to zero, or multiplied by a coefficient less than 1 to reduce the pixel values. After this is performed on all pixels, multiple slice images IMG1 to IMG N .

[0567] (use)

[0568] Fig.70 4 is a block diagram of an object recognition system 400 including an imaging device. Object recognition system 400 includes an imaging device 410 and a processing device 420. Imaging device 410 is imaging device 10 described in Embodiments 6-1 and 6-2, and generates a composite image IMGc.

[0569] The operation processing device 420 is configured to be able to recognize the position and type (category, type) of the object based on the synthesized image IMGc. The operation processing device 420 may include a classifier 422. The operation processing device 420 may be implemented by a combination of a processor (hardware) such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a microcomputer, and a software program executed by the processor (hardware). Alternatively, the operation processing device 420 may be configured only by hardware.

[0570] The classifier 422 is implemented based on the prediction model generated by machine learning, and determines the type (category, type) of the object contained in the input image. The algorithm of the classifier 422 is not particularly limited, but YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Region-based Convolutional Neural Network), SPPnet (Spatial Pyramid Pooling), Faster R-CNN, DSSD (Deconvolution-SSD), Mask R-CNN, etc. can be used, or an algorithm to be developed in the future can be used. The processing device 420 and the image processing device 30 of the camera device 410 can be installed on the same processor or the same FPGA.

[0571] The output of the object recognition system 400 can be used in the light distribution control of the vehicle lamp, and can also be sent to the vehicle-side ECU and used in the automatic driving control.

[0572] Fig.715 is a block diagram of a display system 500 having an imaging device. The display system 500 includes an imaging device 510 and a display 520. The imaging device 510 is the imaging device 10 described in Embodiments 6-1 and 6-2, and generates a composite image IMGc with improved image quality. The display 520 displays the composite image IMGc. The display system 500 may be a digital mirror, or a front view monitor or a rear view monitor for covering a blind spot.

[0573] Fig.72 1 is a diagram showing a car 300 equipped with an object recognition system 400. The car 300 includes headlights 302L and 302R. A part or all of the object recognition system 400 is built into at least one of the headlights 302L and 302R. The headlight 302 is located at the front end of the car body and is the most advantageous location for the strobe camera 20 to detect surrounding objects.

[0574] Fig.73 2 is a block diagram showing a vehicle lamp 200 having an object detection system 210. The vehicle lamp 200 and the vehicle-side ECU 304 together constitute a lamp system 310. The vehicle lamp 200 includes a light source 202, a lighting circuit 204, and an optical system 206. In addition, the vehicle lamp 200 is provided with an object detection system 210. The object detection system 210 corresponds to the above-mentioned object recognition system 400, and includes a strobe camera 20, an image processing device 30, and an operation processing device 40.

[0575] The information related to the object OBJ detected by the calculation processing device 40 can also be used in the light distribution control of the vehicle lamp 200. Specifically, the lamp-side ECU 208 generates an appropriate light distribution pattern based on the information related to the type and position of the object OBJ generated by the calculation processing device 40. The lighting circuit 204 and the optical system 206 operate to obtain the light distribution pattern generated by the lamp-side ECU 208.

[0576] In addition, the information related to the object OBJ detected by the operation processing device 40 can be sent to the vehicle-side ECU 304. The vehicle-side ECU can perform automatic driving based on the information. In object detection, the function of the operation processing device 40 can be installed in the vehicle-side ECU 304.

[0577] The present invention is described using specific language based on the implementation mode, but the implementation mode only represents one aspect of the principle and application of the present invention. In the implementation mode, multiple variations or configuration changes are allowed without departing from the scope of the idea of ​​the present invention as defined by the scope of the claims.

[0578] [Industrial Applicability]

[0579] The present invention relates to strobed cameras.

[0580] [Explanation of Reference Numerals]

[0581] S1 Lighting timing signal

[0582] S2 Photography timing signal

[0583] 10 Object Recognition System

[0584] 12 Visible light camera

[0585] 20 Strobe Camera

[0586] 22 lighting fixtures

[0587] 24A First Image Sensor

[0588] 24B Second image sensor

[0589] 24 Infrared Camera

[0590] 26 Controller

[0591] 40. Operation processing device

[0592] 42 Classifier

[0593] 50 Separation Processing Unit

[0594] 52 First processing unit

[0595] 54 Second processing unit

[0596] 200 Vehicle lighting

[0597] 202 Light Source

[0598] 204 Lighting Circuit

[0599] 206 Optical System

[0600] 300 Cars

[0601] 302 headlamp

[0602] 304 Vehicle side ECU

[0603] 310 Lighting System.

Claims

1. A strobe camera, characterized in that: A gated camera for generating a plurality of slice images corresponding to a plurality of ranges divided in a depth direction, comprising: an illumination device for irradiating a detection light, The first image sensor, a second image sensor, and a controller for controlling the irradiation of the detection light by the illumination device and the timing of photographing by the first image sensor and the second image sensor; The controller controls so that the first image sensor and the second image sensor receive reflected light from different ranges respectively, corresponding to the irradiation of the primary detection light of the lighting device. When the combination of the light emission of the illumination device and the exposure performed by the first image sensor and the second image sensor is referred to as a time slot, the longer the range is, the more time slots are allocated to the range, and each image obtained in each time slot of the range is synthesized to generate the slice image of the range; The total number of time slots required for the plurality of ranges is allocated to the first image sensor and the second image sensor in half, and processing is performed in parallel.

2. The strobe camera according to claim 1, characterized in that The first image sensor is assigned to a relatively close range, The second image sensor is assigned to a relatively long measuring range.

3. The strobe camera according to claim 2, characterized in that The field of view of the first image sensor is wider than the field of view of the second image sensor.

4. The strobe camera according to any one of claims 1 to 3, characterized in that Mounted on a vehicle.

5. The strobe camera according to claim 4, characterized in that The first image sensor is mounted on the right lamp, and the second image sensor is mounted on the left lamp.

6. A car, characterized in that: include: A strobe camera as claimed in any one of claims 1 to 5; as well as A processing device processes the multiple slice images taken by the strobe camera.

7. A vehicle lamp, characterized in that: A strobe camera according to any one of claims 1 to 5.

Images