On-vehicle outside recognition device

Abstract

To provide an on-vehicle outside recognition device capable of reliably recognizing an object located near an imaging part.SOLUTION: An on-vehicle outside recognition device is mounted on a vehicle having an imaging part and a distance measurement part. An arithmetic unit of the on-vehicle outside recognition device detects an object by using a distance measurement result by the distance measurement part, sets a search range for performing model estimation processing on an image picked up by the imaging part by using a detection result of the object, and performs model estimation of the object within the search range from the image obtained by imaging the range from first height higher than a ground point of the object to a second height by using type specification information for specifying a type of the object stored in a prescribed storage unit.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an in-vehicle external environment recognition device. [Background technology]

[0002] When recognizing an object from an image captured by a camera, there are devices that use sensors other than a camera to assist in recognizing the object. For example, Patent Document 1 describes an image processing device that identifies a road surface area using a captured image and distance measurement results by a laser radar. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-143979 A Summary of the Invention [Problem to be solved by the invention]

[0004] The image processing device described in Patent Document 1 may not be able to recognize an object located near the camera, for example, if part of the object deviates from the field of view of the camera or if part of the object is obstructed by an obstruction.

[0005] An object of the present invention is to provide an in-vehicle external environment recognition device that can also recognize objects located in the vicinity of an imaging unit. [Means for solving the problem]

[0006] An in-vehicle external environment recognition device according to one aspect of the present invention is an in-vehicle external environment recognition device equipped with a computing device that is mounted on a vehicle having an imaging unit and a ranging unit, wherein the computing device detects an object using a ranging result by the ranging unit, and uses the object detection result to set a search range in which model estimation processing is performed on an image captured by the imaging unit, and performs model estimation of the object in the search range from an image captured in a range from a first height to a second height higher than the ground point of the object, using type identification information for identifying the type of the object stored in a specified memory unit. Effect of the Invention

[0007] According to the present invention, an object located near the imaging unit can be reliably recognized. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram of a vehicle equipped with an external environment recognition device according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating a hardware configuration of the external environment recognition device according to the first embodiment. [Diagram 3] FIG. 3 is a block diagram illustrating a functional configuration of the external environment recognition device according to the first embodiment. [Figure 4] FIG. 4 is a flowchart of an external environment recognition process performed by the external environment recognition device according to the first embodiment. [Diagram 5] FIG. 5 is an explanatory diagram for explaining a method for determining the vertical angle of view. [Figure 6] FIG. 6 is an explanatory diagram showing the vertical angle of view corresponding to the height of the top end of the search range. [Figure 7] FIG. 7 is an explanatory diagram showing the vertical angle of view corresponding to the height of the bottom end of the search range. [Figure 8] FIG. 8 is an explanatory diagram for explaining a method for determining the horizontal angle of view. [Figure 9] FIG. 9 is an explanatory diagram of a method for setting a search range in a corrected image. [Figure 10]FIG. 10 is an explanatory diagram showing an example of a model configuration when model estimation is performed using a binary discriminator. [Figure 11] FIG. 11 is an explanatory diagram showing an example of a model configuration when performing regression estimation. [Figure 12] FIG. 12 is an explanatory diagram showing an example of a model configuration for solving a multi-value classification problem, in which a model is defined for different classes according to height. [Figure 13] FIG. 13 is an explanatory diagram illustrating information that the search range setting unit stores as learning data for updating the model dictionary. [Figure 14] FIG. 14 is a schematic diagram of a vehicle equipped with an external environment recognition device according to the second embodiment. [Figure 15] FIG. 15 is a block diagram illustrating a functional configuration of an external environment recognition device according to the second embodiment. [Figure 16] FIG. 16 is a flowchart of an external environment recognition process performed by the external environment recognition device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] First Embodiment An in-vehicle external environment recognition device according to a first embodiment of the present invention will be described with reference to Figures 1 to 13. In the following description, the in-vehicle external environment recognition device may be simply referred to as the external environment recognition device.

[0010] FIG. 1 is a schematic diagram of a vehicle equipped with an external environment recognition device according to the first embodiment. The external environment recognition device according to the first embodiment is a device that is equipped in a vehicle 1 and recognizes the external environment of the vehicle 1. In the following description, the vehicle 1 may be referred to as the vehicle itself. In addition to the external environment recognition device, the vehicle 1 is equipped with a distance measuring device 2a, a distance measuring device 2b, and an imaging device 3. In the following description, the distance measuring device 2a and the distance measuring device 2b are collectively referred to as the distance measuring device 2.

[0011] The distance measuring device 2a is a distance measuring device such as a millimeter wave radar or LiDAR (Laser Imaging Detection and Ranging). The distance measuring device 2 is attached to the inside of the front bumper of the vehicle 1. The distance measuring device 2a measures the distance to an object present in the field of view 101 on the front left of the vehicle 1 and the relative speed between the object and the vehicle 1, and outputs distance information indicating the distance and relative speed information indicating the relative speed to the external environment recognition device. The distance measuring device 2b measures the distance to an object present in the field of view 102 on the front right of the vehicle 1 and the relative speed between the object and the vehicle 1, and outputs distance information indicating the distance and relative speed information indicating the relative speed to the external environment recognition device.

[0012] For example, if the distance measuring device 2 is a millimeter wave radar device, the distance measuring device 2 transmits so-called millimeter waves (radio waves with a wavelength of about 1 to 10 mm, in other words, radio waves with a frequency of about 30 to 300 GHz) in various directions, receives the radio waves reflected from various objects in the outside world, and detects the distance and relative speed of the objects in the outside world for each direction based on the time difference between the time the radio waves are transmitted and the time they are received, or the wavelength difference between the transmitted and received radio waves.

[0013] In addition, if the ranging device 2 is a LiDAR, the ranging device 2 emits light such as near-infrared or visible light in various directions, receives the light reflected from various objects in the outside world, detects the time difference between the time of emission and the time of reception of the light and the wavelength difference between the emitted light and the received light, and detects the distance and relative speed for each direction of the objects in the outside world.

[0014] The imaging device 3 is an imaging device including an image sensor such as a CMOS image sensor and an imaging optical system. The imaging device 3 is attached in the cabin of the vehicle 1 facing the front of the vehicle 1. In the example of FIG. 1, the optical axis of the imaging device 3 is approximately aligned with a central axis 100 of the vehicle 1 at 0 degrees in front. The imaging device 3 captures an image of a field of view 103 in front of the vehicle 1 and outputs an imaging signal to the external environment recognition device. In this embodiment, a wide field of view can be ensured by separately installing the distance measuring device 2 and the imaging device 3.

[0015] FIG. 2 is a block diagram showing a schematic hardware configuration of the external recognition device according to the first embodiment. The external recognition device 4 is composed of a computer including a calculation device 11 such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), a non-volatile memory 12 such as a ROM (Read Only Memory), a flash memory, a hard disk drive, a volatile memory 13 called a RAM (Random Access Memory), an input / output interface 14, and other peripheral circuits. These hardware components work together to operate software and realize multiple functions. The external recognition device 4 may be composed of one computer or multiple computers. In addition, the calculation device 11 may be an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like.

[0016] The non-volatile memory 12 stores programs capable of executing various calculations. In other words, the non-volatile memory 12 is a storage medium (storage device) capable of reading programs that realize the functions of this embodiment. The volatile memory 13 is a storage medium (storage device) that temporarily stores the results of calculations by the calculation device 11 and signals input from the input / output interface 14. The calculation device 11 is a device that loads the programs stored in the non-volatile memory 12 into the volatile memory and executes calculations, and performs predetermined calculation processing on data taken from the input / output interface 14, the non-volatile memory 12, and the volatile memory 13 in accordance with the programs.

[0017] The input / output interface 14 is connected to each of the distance measuring device 2 and the imaging device 3. An input section of the input / output interface 14 converts signals input from various devices (such as the distance measuring device 2 and the imaging device 3) into data that can be calculated by the arithmetic device 11. In addition, an output section of the input / output interface 14 generates an output signal according to the calculation result in the arithmetic device 11, and outputs the signal to the various devices.

[0018] 3 is a block diagram showing a functional configuration of the external environment recognition device according to the first embodiment. The external environment recognition device 4 includes a noise removal unit 41, an object detection unit 42, a geometric correction unit 43, a search range setting unit 44, and a model estimation unit 45.

[0019] The noise removal unit 41 removes what is considered to be noise from the distance information and relative speed information for each direction detected by the distance measuring device 2. Here, what is considered to be noise is, for example, distance information and relative speed information based on millimeter waves or light that are not directly reflected from an object. Or, distance information and relative speed information based on millimeter waves or light reflected by an object that is not expected to be a detection target object. An object that is not expected to be a detection target object is, for example, a stationary object that is always present, such as a road surface. For example, when the vehicle 1 is traveling, a relative speed equal to the traveling speed of the vehicle 1 is detected from the millimeter waves or light reflected from the road surface. When such a relative speed is detected, the noise removal unit 41 takes into account information such as reflectivity and determines that the detected object is the road surface, and removes the distance information and relative speed information related to the object.

[0020] The object detection unit 42 detects an object existing in the outside world based on the distance information and relative speed information after the noise removal unit 41 has performed the removal process. The object detection unit 42 detects the distance information and relative speed information detected at a plurality of locations as a single object, taking into account information such as reflectance. For example, if the distance measuring device 2 detects an object at a certain distance in a certain direction, and further detects an object at approximately the same distance in a direction slightly shifted from that direction, if the relative speed, reflectance, etc. are similar, the two detected objects are considered to be the same object. Based on this concept, the object detection unit 42 detects an actual object from the distance information and relative speed information for each direction. The object detection unit 42 infers the center position of the object, and sets the inferred center position as the reference position of the object. Note that a position other than the center position of the object, such as the road contact point of the object, may be set as the reference position. The object detection unit 42 sets the reference position of the detected object based on a vehicle coordinate system based on a predetermined position of the vehicle 1 (for example, the center position of the vehicle 1). If necessary, the object detection unit 42 performs conversion from the coordinate system based on the position of the distance measuring device 2 to the vehicle coordinate system based on the predetermined position of the vehicle 1.

[0021] The geometric correction unit 43 performs geometric correction of the imaging signal output by the imaging device 3 based on a geometric table stored in advance in the non-volatile memory 12. The geometric table includes the camera parameters of the imaging device 3. The camera parameters include, for example, the focal length of the optical system of the imaging device 3, distortion information, and the installation position of the imaging device 3 in the vehicle coordinate system. The geometric correction unit 43 performs a well-known perspective transformation using the geometric table, and outputs a corrected image in which the imaging signal has been subjected to geometric correction. The corrected image associates the two-dimensional coordinates of each pixel in the imaging signal captured by the imaging device 3 with the vehicle coordinate system. By using the corrected image, if only the depth of each pixel (i.e., the distance from the vehicle 1 to the object corresponding to each pixel) can be identified, the position of the object corresponding to that pixel in the vehicle coordinate system can be identified.

[0022] The search range setting unit 44 sets a part of the entire corrected image as a search range in which model estimation is performed by the model estimation unit 45. The search range setting unit 44 specifies the position of the object detected by the object detection unit 42 on the imaging signal geometrically corrected by the geometric correction unit 43, and sets the range including that position as a search range corresponding to the object.

[0023] The model estimation unit 45 performs model estimation for the corrected image using a model dictionary stored in advance in the non-volatile memory 12. The model dictionary includes type identification information for identifying the type of object. The model estimation unit 45 identifies whether the search range set by the search range setting unit 44 matches a model in the model dictionary, and performs regression estimation of coordinates that match the model, thereby identifying the type of object and the coordinates of the object captured in the search range. The model estimation unit 45 performs identification and regression estimation based on a model dictionary such as a weighting coefficient learned by machine learning. In this embodiment, it is assumed that the model dictionary is created in advance by machine learning, but the external environment recognition device 4 may dynamically create and update the model dictionary.

[0024] The result of the model estimation by the model estimation unit 45 is output from the external environment recognition device 4 to another device in the vehicle 1. For example, the result of the model estimation is output to another control device that predicts a collision between the vehicle 1 and another object. In such a control device, by using the result of the model estimation to predict a collision with the vehicle 1, it is possible to issue a warning to the driver of the vehicle 1 when a possibility of a collision between the vehicle 1 and an external object is predicted, or to perform vehicle control such as automatically applying the brakes to avoid a collision.

[0025] FIG. 4 is a flowchart of the external environment recognition process by the external environment recognition device 4 according to the first embodiment. In step S100, distance information and relative speed information are input from the distance measuring device 2 to the calculation device 11. In step S110, the noise removal unit 41 removes noise from the distance information and relative speed information input in step S100. In step S120, the object detection unit 42 detects an object using the distance information and relative speed information. In step S130, an imaging signal is input from the imaging device 3 to the calculation device 11. In step S140, the geometric correction unit 43 performs geometric correction on the imaging signal input in step S130 using a geometric table stored in the non-volatile memory 12, and outputs a corrected image. In step S150, the search range setting unit 44 executes a process of specifying the vertical angle of view and the horizontal angle of view, which will be described later. In step S160, the search range setting unit 44 sets a search range for the corrected image output in step S140 using the vertical angle of view and the horizontal angle of view specified in step S150. In step S170, the search range setting unit 44 stores information for additional learning in the non-volatile memory 12. The information for additional learning is, for example, information that becomes an input (learning data) when updating the model dictionary by machine learning. Such machine learning may be performed outside the external environment recognition device 4, or may be performed by the arithmetic device 11 during operation of the external environment recognition device 4. Examples of the information to be stored include information indicating the position of the search range set in step S150 (i.e., information such as the height of the upper end and the height of the lower end, the positions of the left and right ends, the vertical angle of view, and the horizontal angle of view, which will be described later), a partial image obtained by cutting out an area corresponding to the search range from the corrected image (i.e., luminance information and color information within the search range), and information regarding the position of an object present within the search range (i.e., information indicating distance information and height from the road surface, etc.). In step S180, the model estimation unit 45 performs model estimation for the corrected image based on the model dictionary and the search range set in step S160. In step S190, the model estimation unit 45 outputs the result of the model estimation performed in step S180 to the outside of the external environment recognition device 4.

[0026] A detailed description will be given of the process of specifying the vertical angle of view and the horizontal angle of view by the search range setting unit 44. The search range setting unit 44 specifies the vertical angle of view and the horizontal angle of view of the search range in the imaging device 3 in order to set a search range for performing model estimation.

[0027] FIG. 5 is an explanatory diagram for explaining a method for specifying the vertical angle of view. In the following description, it is assumed that the model to be estimated by the model estimation is a pedestrian. In FIG. 5, the pedestrian 50 to be estimated is located in front of the vehicle 1, at a distance 53 from the image capture device 3 in the front direction of the vehicle 1. The vertical distance of the image capture device 3 from the road surface is the installation height 54. When specifying the vertical angle of view, the height 51 of the upper end and the height 52 of the lower end of the search range are determined in advance. Although there may be individual differences in the size of the model to be estimated, it is desirable that most of the individuals are included within the range from the height 51 of the upper end to the height 52 of the lower end, and that the range is not too large. If many individuals are not included within the range, the accuracy of the model estimation decreases. Moreover, if the range is made too wide, the time required for the model estimation may increase and the estimation accuracy may decrease.

[0028] The top height 51 is the height that is set as the upper limit when learning the model. For example, if the model is a pedestrian, it is considered that the majority of pedestrians can be covered by setting the top height 51 to about 180 cm to 250 cm. Similarly, for objects other than pedestrians, it is advisable to assume a height that is the upper limit of the height of the individual, or a height that is several times the standard deviation from the average of the object type to be modeled, which includes the top ends of most individuals.

[0029] In addition, in model estimation, it is possible to identify the object type and specify the road contact point even if the entire object is not necessarily within the search range. Therefore, the upper end height 51 may be set to the average height in the area where the external environment recognition device 4 is operated plus several times the standard deviation of the height in the area. If the height of the model is very high and does not fit within the upper end of the vertical angle of view of the imaging device 3, the height is set to the height of the position seen at the upper end of the vertical angle of view of the imaging device 3 at the nearest distance to be recognized, as with the lower end height 52 described later.

[0030] The lower end height 52 is the height of the object, whichever is higher, between the lowest position visible when an object to be recognized (e.g., a pedestrian) is shielded by the bonnet of the vehicle 1 or the position captured at the lower end of the vertical angle of view of the imaging device 3, when the object is present at the nearest distance desired to be recognized. The reason why the contact point with the road surface is not simply set as the lower end height 52 is that when an object exists near the imaging device 3, the lower end of the object may be shielded by the bonnet of the vehicle 1 or may be outside the imaging range of the imaging device 3, in which case the object will not be captured at the position corresponding to the contact point in the corrected image. The lower end height 52 is determined based on the installation position of the imaging device 3, the imaging range of the imaging device 3, the type of vehicle 1 (bonnet height), etc.

[0031] It is desirable to provide a margin for the upper end height 51 and the lower end height 52. In other words, it is desirable to set the upper end height 51 at a slightly higher position and the lower end height 52 at a slightly lower position. By doing so, even when the angle of view cannot be obtained as designed due to shaking of the imaging device 3 or manufacturing variations, a search range can be set in which the object to be the subject of model estimation is reliably captured in the corrected image, and recognition accuracy can be improved.

[0032] In model estimation, it is determined whether an object in a search range set in a corrected image based on the position of an object detected by the distance measuring device 2 corresponds to a model, and also where in the image the object corresponding to the model is located, including its width and height that cannot be determined from distance information alone. The model dictionary used in model estimation is a dictionary of coefficients and the like of a model to be identified that has been created in advance by machine learning or the like. The search range setting unit 44 sets a search range according to the height 51 of the upper end and the height 52 of the lower end of the model assumed when determining the installation position of the imaging device 3 or when additionally learning the model to be identified.

[0033] 6 is an explanatory diagram showing a vertical angle of view corresponding to the height 51 of the top end of a search range. First, a distance 55 to the top end height 51 is calculated based on the height 54 of the imaging device 3. Then, by applying a trigonometric function using the distance 53 from the imaging device 3 to the pedestrian 50 and the distance 55 to the top end height 51 based on the height 54 of the imaging device 3, a vertical angle of view 56 to the top end height 51 is calculated when an axis extending from the imaging device 3 parallel to the road surface is set to 0 degrees.

[0034] 7 is an explanatory diagram showing a vertical angle of view corresponding to a height 52 of the bottom end of a search range. First, a distance 57 to the bottom end height 52 is calculated based on a height 54 of the imaging device 3. Then, by applying a trigonometric function using a distance 53 from the imaging device 3 to a pedestrian 50 and a distance 57 to the bottom end height 52 based on the height 54 of the imaging device 3, a vertical angle of view 58 to the bottom end height 52 is calculated when an axis extending from the imaging device 3 parallel to the road surface is set to 0 degrees. The vertical angle of view of the search range can be specified by adding a vertical angle of view 56 to the top end height 51 shown in FIG. 6 and a vertical angle of view 58 to the bottom end height 52 shown in FIG. 7 together.

[0035] FIG. 8 is an explanatory diagram for explaining a method for specifying the horizontal angle of view. When specifying the horizontal angle of view, a search range width 60 is determined in advance. Although individual differences in the size of the model to be estimated are conceivable, it is desirable that most of the individuals are included within the range of the search range width 60, and that the range is not too large. If many individuals are not included within the range, the accuracy of model estimation decreases. Moreover, if the range is made excessively wide, the time required for model estimation may increase and the estimation accuracy may decrease.

[0036] The search range width 60 is set to be large enough as the width of an object that appears in a corrected image when performing model identification and regression estimation by model estimation. For example, if the model to be estimated is a pedestrian, the search range width 60 is set to about 100 cm. When a wide object is used as a model, the search range width 60 is set accordingly wide, and when a narrow object is used as a model, the search range width 60 is set accordingly narrow, thereby making it possible to reduce the search range for model estimation and the amount of calculations.

[0037] First, the right end and the left end of the search range are identified using a predetermined search range width 60 with respect to the center position of the object detected by the object detection unit 42. Specifically, the right end of the search range is determined as a position moved to the right by half the search range width 60 from the center position of the object, and the left end of the search range is determined as a position moved to the left by half the search range width 60 from the center position of the object. Then, by applying trigonometric functions using the right and left ends of the search range and the distance 53 from the imaging device 3 to the pedestrian 50, a horizontal angle of view 61 to the left end of the search range and a horizontal angle of view 62 to the right end of the search range when an axis extending from the imaging device 3 in a direction in front of the imaging device 3 is set to 0 degrees are obtained.

[0038] Fig. 9 is an explanatory diagram of a method for setting a search range in a corrected image. Fig. 9 shows an example of a corrected image, that is, a perspective projection image 63 mapped to a perspective projection, and a search range 64 set thereon. Note that, even in the case of other projection planes, the vertical angle of view of the specified upper and lower ends and the horizontal angle of view of the specified left and right ends can be converted into image coordinates according to the formula for the height of the optical axis center.

[0039] The search range in the corrected image is set according to the following formulas (1) to (4): where xleft is the horizontal image coordinate of the left end of the search range, xright is the horizontal image coordinate of the right end of the search range, ytop is the vertical image coordinate of the top end of the search range, ybottom is the vertical image coordinate of the bottom end of the search range, f is the focal length of the imaging section of the imaging device, ph is the horizontal pixel pitch of the image sensor, pv is the vertical pixel pitch of the image sensor, θleft is the horizontal angle of view corresponding to the distance to the left end of the search range (corresponding to the horizontal angle of view 61 in FIG. 8), θright is the horizontal angle of view corresponding to the distance to the right end of the search range (corresponding to the horizontal angle of view 62 in FIG. 8), θ1 is the vertical angle of view corresponding to the height of the top end of the search range (corresponding to the vertical angle of view 56 in FIG. 6), θ2 is the vertical angle of view corresponding to the height of the bottom end of the search range (corresponding to the vertical angle of view 58 in FIG. 7), wimage is the image width, and himage is the image height.

[0040]

number

[0041] Note that there will be variation between the calculated object position and its position on the image due to variations in the object for model estimation, the attitude of the imaging device 3, distance measurement errors by the distance measuring device 2, etc. For this reason, it is desirable to provide some margin in the width and height of the left and right ends and the top and bottom ends of the search range calculated by the above formulas (1) to (4).

[0042] When performing regression estimation as described below, if the top and bottom ends or the left and right ends are not included in the search range in regression estimation, accurate regression estimation is hindered, so it is desirable to make the margins of width and height larger. Also, when classifying by height as described below, in order to accurately estimate the cut-out parts of the pedestrian's body that are captured in the search range, it is better to expand the width and height based on the physical width and height rather than the number of pixels on the image. By doing so, the parts of the body that are captured in the search range for a specific height do not change regardless of the distance, making it possible to estimate the height accurately.

[0043] FIG. 10 is an explanatory diagram showing an example of a model configuration when model estimation is performed using a binary classifier. An example is shown below in which the characteristics of a pedestrian are defined as a model and a binary classification of whether or not the object is a pedestrian is performed. In this case, a large number of images in which pedestrian parts have been cut out are machine-learned, and a pedestrian model is entered as a coefficient in a model dictionary. The external environment recognition device 4 uses this model dictionary to classify pedestrians from the corrected image. In this example, a binary example is used, but by defining a multi-class model with three or more values, objects other than pedestrians, such as vehicles and cyclists, can also be classified.

[0044] For example, as shown in image 71 and image 72 in FIG. 10, a large number of images in which pedestrians are captured in the external environment where the external environment recognition device 4 is actually used are prepared. Then, images in which pedestrians are cut out from these images are used as a pedestrian class for machine learning. Similarly, as shown in image 73 in FIG. 10, a large number of images showing the background of a scene in which the pedestrian may exist are prepared. These images are used as a background class for machine learning.

[0045] The pedestrian class images (e.g., image 71, image 72) and background class images (e.g., image 73) prepared in this way are used as inputs to perform machine learning using, for example, a CNN (Convolution Neural Network), and the co-occurring image features of the images used for learning are entered as coefficients into a model dictionary. If the features described in the model dictionary are similar (co-occur) to an image appearing in the search range of model estimation in the corrected image, the corresponding model is output as the identification result. For example, if they co-occur with the pedestrian class, it is identified that a pedestrian is included in the search range of the image, and if they do not co-occur, it is identified that an object other than a pedestrian is included in the search range of the image. By performing machine learning on various expected variations to create an identification model dictionary, it is possible to identify the type of object with a high identification rate.

[0046] FIG. 11 is an explanatory diagram showing an example of a model configuration when performing regression estimation. Below, an example of performing regression estimation of a pedestrian by defining the characteristics of a pedestrian as a model is shown. In regression estimation, the coordinates of a specific position (e.g., left, right, top, bottom) of a modeled object within a search range are estimated. Images in which the model appears (e.g., images 74, 75, and 76 shown in FIG. 11) and the coordinates of a specific position of the model (e.g., left, right, top, bottom, e.g., images 78, 79, and 790) are given as training data, and machine learning is performed using, for example, CNN. At this time, an image in which the model to be estimated does not appear (e.g., image 77 shown in FIG. 11) is also input as training data.

[0047] FIG. 12 is an explanatory diagram showing an example of the configuration of a model for solving a multi-value classification problem, in which a model is defined for different classes according to height. For example, the height of the upper end of the search range is set to 200 cm, and the height of the lower end is set to 50 cm, and images of pedestrians captured within that range are cut out and trained into classes according to height. FIG. 12 shows training images 80 of pedestrians with heights of around 200 cm, training images 81 of pedestrians with heights of around 180 cm, training images 82 of pedestrians with heights of around 160 cm, and training images 83 of pedestrians with heights of around 140 cm. In this way, a large number of training images divided into four classes from 200 cm to 140 cm, in increments of 20 cm, are prepared and trained as different classes.

[0048] By dividing the classes by height in this way, it is possible to estimate not only the type of object but also its height and height at the same time. Note that in this example, the classes are divided every 20 cm, with a maximum of 200 cm and a minimum of 140 cm, but it is possible to divide into more classes to increase the resolution and range of distinguishable heights.

[0049] Fig. 13 is an explanatory diagram illustrating an example of information stored as learning data for updating the model dictionary by the search range setting unit 44. As described in step S170 of Fig. 4, the search range setting unit 44 stores, for example, a partial image obtained by cutting out a portion corresponding to the search range from the corrected image (i.e., a portion showing the model), information indicating where in the corrected image the heights of the upper and lower ends of the search range correspond, distance information, height information from the road surface, and the like.

[0050] For example, as shown in Fig. 13, it is assumed that the imaging device 3 captures an image 84. At this time, the search range setting unit 44 sets an area in which a pedestrian, which is a model of the estimation target, is captured as the search range. The search range setting unit 44 cuts out a partial image 85 corresponding to the search range from the image 84 and stores it. Depending on the number of pixels of the image sensor of the imaging device 3, the data size of the image 84 may become large. Therefore, it is desirable to reduce the data size of the image for additional learning and save storage area by cutting out an image in the vicinity of an object to be model-defined based on information on the detected object, as in the partial image 85.

[0051] Search range setting unit 44 stores partial image 85 together with distance information of pedestrians contained therein, acquired by distance measuring device 2. Distance information 86 illustrated in Fig. 13 is an illustration of the results of distance measurement by distance measuring device 2 arranged as in image 84. What is actually stored is partial distance information 87, which is an area corresponding to the search range cut out from distance information 86, similar to partial image 85 for image 84. As in the case of storing partial image 85, storing partial distance information 87 instead of distance information 86 can save storage area.

[0052] The search range setting unit 44 stores the partial image 85 and the partial distance information 87 together with the height information of the object appearing therein. The height information 88 illustrated in FIG. 13 is height information calculated for each vertical and horizontal angle of view in the image based on the image captured by the imaging device 3, the distance information acquired by the distance measuring device 2, and the information of the object detected by the object detection unit 42. What is actually stored is partial height information 89 obtained by cutting out an area corresponding to the search range from the height information 88, similar to the partial image 85 for the image 84. As in the case of storing the partial image 85, storing the partial height information 89 instead of the height information 88 can save storage space.

[0053] In addition to the various information described above, the result of the model estimation by the model estimator 45 may be stored for additional learning. In this case, step S170 in the flowchart shown in FIG. 4 needs to be performed after the model estimation process in step S180.

[0054] In this way, when the heights of the upper and lower ends of the search range are set according to the specifications of the vehicle 1 and the specifications and mounting position of the imaging device 3, it becomes possible to freely extract and learn learning images for machine learning by storing the image, distance information, height information, etc., for which a model is to be defined, as shown in Fig. 13. This eliminates the need to collect learning images, which is an enormously time-consuming task, for each vehicle, and thus reduces the development labor.

[0055] According to the above-described first embodiment, the following advantageous effects are achieved.

[0056] (1) The computing device 11 detects an object using the distance measurement result by the distance measurement device 2 (distance measurement unit), sets a search range in which model estimation processing is performed on an image captured by the imaging device 3 (imaging unit) using the object detection result, and performs model estimation of the object in the search range from an image captured in a range from a height (first height) at the top end of the search range higher than the ground point of the object to a height (second height) at the bottom end of the search range using type identification information (model dictionary) for identifying the type of object stored in the non-volatile memory 12 (predetermined storage unit).By doing so, for example, a part of the object does not deviate from the field of view of the imaging device 3 or a part of the object is not blocked by an obstruction, and an object located near the imaging device (camera) 3 can be reliably recognized.

[0057] (2) The arithmetic unit 11 uses a discrimination model for discriminating the type of object, which makes it possible to discriminate nearby objects.

[0058] (3) The arithmetic unit 11 uses a regression model that performs regression estimation of object position information, making it possible to estimate detailed position information of nearby objects.

[0059] (4) The computing device 11 uses a classification model that classifies the class of an object according to its height. This makes it possible to simultaneously estimate not only the type of object but also its height (body height) even when the area around the target object is not visible.

[0060] (5) The calculation device 11 sets the height of the top end of the search range (first height) according to the height of the top end of the object defined in the model estimation. This makes it possible to increase the proportion of the model in the image range of the search range for model estimation, thereby making it possible to save data capacity.

[0061] (6) The computing device 11 sets the height of the bottom end of the search range (second height) according to the height of the bottom end of a nearby object that is not occluded by the bonnet of the vehicle 1 and appears in the image. This makes it possible to increase the proportion of the model in the image range of the search range for model estimation while enabling model estimation in the vicinity by not using the features of the occluded part, thereby saving data capacity.

[0062] (7) The computing device 11 stores an image of the detected object and distance information corresponding to the image. In this way, it is possible to perform model estimation in the vicinity without using the features of the occluded part, while increasing the proportion of the model in the image range of the search range for model estimation, thereby saving data capacity.

[0063] <Second embodiment> An external environment recognition device according to a second embodiment of the present invention will be described with reference to Fig. 14 to Fig. 16. Note that the same reference symbols are used for configurations that are the same as or equivalent to those described in the first embodiment, and differences will be mainly described.

[0064] FIG. 14 is a schematic diagram of a vehicle equipped with an external environment recognition device according to a second embodiment, similar to FIG. 1. An imaging device 5 is installed in the vehicle 1 instead of the distance measuring device 2 and the imaging device 3. The imaging device 5 is a so-called stereo camera, and incorporates an imaging unit 5a provided on the left side and an imaging unit 5b provided on the right side. The imaging unit 5a captures an image of a field of view 201 on the left side of the vehicle 1, and outputs an imaging signal to the external environment recognition device. The imaging unit 5b captures an image of a field of view 202 on the right side of the vehicle 1, and outputs an imaging signal to the external environment recognition device. The imaging unit 5a and the imaging unit 5b are installed at a known distance apart. This distance may be fixed or variable. The external environment recognition device of this embodiment is configured to be able to grasp this distance.

[0065] Since there is a parallax between the imaging units 5a and 5b, as is well known, distance measurement can be performed by utilizing this parallax. In other words, the imaging device 5 can output an image signal by imaging and obtain distance information by distance measurement using the imaging units 5a and 5b. The area where the field of view 201 and the field of view 202 overlap is the stereo field of view, which is an area where distance measurement can be performed by utilizing the parallax.

[0066] 15 is a block diagram similar to FIG. 3, which is a schematic diagram showing the functional configuration of the external environment recognition device according to the second embodiment. The external environment recognition device 204 includes an object detection unit 242, a geometric correction unit 243, and a disparity calculation unit 246 instead of the noise removal unit 41, the object detection unit 42, and the geometric correction unit 43.

[0067] The geometric correction unit 243 performs a geometric correction process on the imaging signal output from the imaging unit 5a and the imaging signal output from the imaging unit 5b, similarly to the geometric correction unit 43 according to the first embodiment. The geometric table of this embodiment is a parallelized geometric table in which epipolar lines are arranged horizontally so that the image of one imaging unit 5a and the image of the other imaging unit 5b are shifted horizontally by a parallax according to the depth distance, so that it is convenient for the parallax calculation unit 246 to calculate the parallax. This geometric table is created, for example, in accordance with a perspective projection plane perpendicular to the parallel optical axes of the base points of the two imaging units 5a and 5b.

[0068] The parallax calculation unit 246 calculates the parallax of the same object using a pair of corrected images that have been geometrically corrected by the geometric correction unit 243. Specifically, it performs so-called stereo matching processing to identify which pixel of the corrected image of one imaging unit 5a corresponds to which pixel of the corrected image of the other imaging unit 5b (whether they correspond to the same object), and outputs a parallax image in which the parallax for each pixel is arranged. For example, a matching method such as SAD (Sum of Absolute Differences) can be used for stereo matching.

[0069] The object detection unit 242 detects objects that appear in the disparity image and the corrected image. For example, the object detection unit 242 detects three-dimensional objects by clustering objects at the same distance using the U-Disparity image.

[0070] Fig. 16 is a flowchart of the external environment recognition process by the external environment recognition device 204 according to the second embodiment, which is similar to Fig. 4. In the flowchart of Fig. 16, the processes of steps S200 to S230 are executed instead of the processes of steps S100 to S140 in the flowchart of Fig. 4.

[0071] In step S200, a pair of imaging signals are input from the imaging device 5 to the calculation device 11. In step S210, the geometric correction unit 243 performs geometric correction on the pair of imaging signals input in step S200 using a geometric table stored in the nonvolatile memory 12, and outputs a pair of corrected images. In step S220, the disparity calculation unit 246 performs disparity calculation on the pair of corrected images output in step S210 using the geometric table stored in the nonvolatile memory 12, and outputs a disparity image. In step S230, the object detection unit 242 detects an object using the pair of corrected images and the disparity image (which has a disparity according to the distance and thus corresponds to distance information).

[0072] In addition, since the imaging device 5 is attached to the vehicle 1, which is a moving body, and captures images while moving, it is necessary for the imaging units 5a and 5b to capture images almost simultaneously. If there is a time difference between capturing images by the imaging units 5a and 5b, the vehicle 1 moves and the object captured in the image moves, resulting in a large error included in the parallax value required for distance measurement.

[0073] According to the above-described second embodiment, the following advantageous effects are achieved.

[0074] (1) The external environment recognition device 204 calculates the distance to an object using the imaging device 5, which is a stereo camera. With this configuration, it is no longer necessary to separately install the imaging device and the distance measuring device, and it is possible to reduce restrictions on the design of the vehicle 1. In addition, the number of parts is reduced, and it is no longer necessary to separately align the distance measuring device, so the assembly cost of the vehicle 1 is reduced.

[0075] The following modified examples are also within the scope of the present invention, and it is possible to combine the configurations shown in the modified examples with the configurations described in the above-mentioned embodiments, to combine the configurations described in the different embodiments above, or to combine the configurations described in the different modified examples below.

[0076] <Variation 1> The number of distance measuring devices 2 and imaging devices 3 may be different from the number shown in FIG. 1. For example, two or more imaging devices 3 may be installed. Also, only one distance measuring device 2 may be installed, or three or more distance measuring devices 2 may be installed. When multiple distance measuring devices 2 are present, the search range setting unit 44 may set a search range for each of the distance measuring devices 2. In this case, multiple search ranges that are considered to correspond to the same object may be integrated into one search range to reduce the number of times model estimation is performed. In this way, the delay until the result of model estimation is output can be reduced.

[0077] <Variation 2> The distance measuring device 2 and the imaging device 3 may be built into the external environment recognition device 4. In other words, the functional units corresponding to the distance measuring device 2 and the imaging device 3 may be provided inside the external environment recognition device 4, rather than being provided outside the external environment recognition device 4. The same applies to the imaging device 5 according to the second embodiment. In this way, the functional units corresponding to the distance measuring device 2 and the imaging device 3 can be connected to the arithmetic device 11 with a faster and lower delay interconnect, and the latency caused by the transfer of the imaging signal and the distance information can be reduced.

[0078] <Variation 3> For model estimation, data other than the model dictionary may be used. For example, weighting coefficients of a DNN (Deep Neural Network) may be stored in the non-volatile memory 12, and model estimation may be performed by a so-called deep learning method. Also, a template of the model may be stored, and model estimation may be performed by a so-called template matching method.

[0079] Although the embodiments of the present invention have been described above, the above-mentioned embodiments merely show some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above-mentioned embodiments. [Explanation of symbols]

[0080] 1...vehicle, 2, 2a, 2b...range measuring device, 3, 5...imaging device, 4, 204...external environment recognition device, 5a, 5b...imaging unit, 11...arithmetic unit, 12...non-volatile memory, 13...volatile memory, 14...input / output interface, 41...noise removal unit, 42, 242...object detection unit, 43, 243...geometric correction unit, 44...search range setting unit, 45...model estimation unit, 246...parallax calculation unit

Claims

1. An in-vehicle external environment recognition device including a computing device, the device being mounted on a vehicle having an imaging unit and a distance measuring unit, The computing device includes: Detecting an object using a distance measurement result by the distance measurement unit; Using the object detection result, a search range is set in which a model estimation process is performed on the image captured by the imaging unit; using type identification information for identifying a type of the object stored in a predetermined storage unit, performing a model estimation of the object in the search range from an image captured in a range from a first height higher than a grounding point of the object to a second height; Vehicle-mounted external world recognition device.

2. The in-vehicle external environment recognition device according to claim 1, The computing device is an in-vehicle external environment recognition device that uses an identification model to identify the type of the object.

3. The in-vehicle external environment recognition device according to claim 1, The computing device is an in-vehicle external environment recognition device that uses a regression model to perform regression estimation of the object's position information.

4. The in-vehicle external environment recognition device according to claim 1, The computing device is an in-vehicle external environment recognition device that uses a discrimination model to discriminate classes according to the height of the object.

5. The in-vehicle external environment recognition device according to claim 1, The computing device is an in-vehicle external environment recognition device that sets a first height according to the height of the upper end of the object defined by the model estimation.

6. The in-vehicle external environment recognition device according to claim 1, The computing device is an in-vehicle external environment recognition device that sets the second height according to the height of the bottom end of a nearby object that is not obstructed by the vehicle hood and appears in the image.

7. The in-vehicle external environment recognition device according to claim 1, The computing device is an in-vehicle external environment recognition device that stores images of detected objects and distance information corresponding to the images.

Images