Onboard external environment recognition device

Abstract

This onboard external environment recognition device is mounted on a vehicle having an imaging unit and a distance measurement unit. A computation device of the onboard external environment recognition device: detects an object using a distance measurement result (S100) obtained by a distance measurement unit (S120); sets, by using the object detection result, a search range for performing a model estimation process on an image (S130) captured by the imaging unit (S160); and, by using type identification information for identifying the type of the object, the type identification information being stored in a prescribed storage unit, performs model estimation of an object in the search range from an image in which a range from a first height higher than a ground point of the object to a second height is captured (S180).

Description

In-vehicle external recognition device

[0001] The present invention relates to an in-vehicle external environment recognition device.

[0002] When recognizing an object from an image captured by a camera, there are devices that use sensors other than the camera to assist in recognizing the object. For example, Patent Document 1 describes an image processing device that identifies a road surface area using a captured image and distance measurement results obtained by a laser radar.

[0003] JP 2015-143979 A

[0004] The image processing device described in Patent Document 1 may not be able to recognize an object located near the camera, for example, if part of the object deviates from the camera's field of view or if part of the object is obstructed by an obstruction.

[0005] An object of the present invention is to provide an in-vehicle external environment recognition device that can also recognize objects located near an imaging unit.

[0006] An in-vehicle external environment recognition device according to one aspect of the present invention is an in-vehicle external environment recognition device equipped with a computing device that is mounted on a vehicle having an imaging unit and a ranging unit, wherein the computing device detects an object using the ranging results from the ranging unit, and uses the object detection results to set a search range in which model estimation processing is performed on the image captured by the imaging unit, and performs model estimation of the object in the search range from an image captured in a range from a first height to a second height that is higher than the ground contact point of the object using type identification information for identifying the type of the object stored in a specified memory unit.

[0007] According to the present invention, an object located near the imaging unit can be reliably recognized.

[0008] FIG. 1 is a schematic diagram of a vehicle equipped with an external environment recognition device according to a first embodiment. FIG. 2 is a block diagram schematically illustrating the hardware configuration of the external environment recognition device according to the first embodiment. FIG. 3 is a block diagram schematically illustrating the functional configuration of the external environment recognition device according to the first embodiment. FIG. 4 is a flowchart of external environment recognition processing by the external environment recognition device according to the first embodiment. FIG. 5 is an explanatory diagram illustrating a method for determining a vertical angle of view. FIG. 6 is an explanatory diagram illustrating a vertical angle of view corresponding to the height of the upper end of a search range. FIG. 7 is an explanatory diagram illustrating a vertical angle of view corresponding to the height of the lower end of a search range. FIG. 8 is an explanatory diagram illustrating a method for determining a horizontal angle of view. FIG. 9 is an explanatory diagram illustrating a method for setting a search range in a corrected image. FIG. 10 is an explanatory diagram illustrating an example of a model configuration when model estimation is performed using a binary classifier. FIG. 11 is an explanatory diagram illustrating an example of a model configuration when regression estimation is performed. FIG. 12 is an explanatory diagram illustrating an example of a model configuration for solving a multi-value classification problem, in which models are defined into different classes according to height. Fig. 13 is an explanatory diagram illustrating information stored by a search range setting unit as learning data for updating a model dictionary. Fig. 14 is a schematic diagram of a vehicle equipped with an external environment recognition device according to a second embodiment. Fig. 15 is a block diagram schematically showing the functional configuration of the external environment recognition device according to the second embodiment. Fig. 16 is a flowchart of external environment recognition processing by the external environment recognition device according to the second embodiment.

[0009] <First embodiment> An in-vehicle external environment recognition device according to a first embodiment of the present invention will be described with reference to Figures 1 to 13. In the following description, the in-vehicle external environment recognition device may be simply referred to as an external environment recognition device.

[0010] FIG. 1 is a schematic diagram of a vehicle equipped with an external environment recognition device according to the first embodiment. The external environment recognition device according to the first embodiment is equipped in a vehicle 1 and recognizes the external environment of the vehicle 1. In the following description, the vehicle 1 may be referred to as the host vehicle. In addition to the external environment recognition device, the vehicle 1 is equipped with a ranging device 2a, a ranging device 2b, and an imaging device 3. In the following description, the ranging device 2a and the ranging device 2b are collectively referred to as the ranging device 2.

[0011] The ranging device 2a is a ranging device such as a millimeter-wave radar or LiDAR (Laser Imaging Detection and Ranging). The ranging device 2 is attached to the inside of the front bumper of the vehicle 1. The ranging device 2a measures the distance to an object present in the field of view 101 on the front left of the vehicle 1 and the relative speed between the object and the vehicle 1, and outputs distance information indicating the distance and relative speed information indicating the relative speed to the external environment recognition device. The ranging device 2b measures the distance to an object present in the field of view 102 on the front right of the vehicle 1 and the relative speed between the object and the vehicle 1, and outputs distance information indicating the distance and relative speed information indicating the relative speed to the external environment recognition device.

[0012] For example, if the distance measuring device 2 is a millimeter wave radar device, the distance measuring device 2 transmits so-called millimeter waves (radio waves with a wavelength of about 1 to 10 mm, in other words, radio waves with a frequency of about 30 to 300 GHz) in various directions, receives the radio waves reflected from various objects in the outside world, and detects the distance and relative speed of the objects in the outside world for each direction based on the time difference between the time the radio waves are transmitted and the time they are received, or the wavelength difference between the transmitted and received radio waves.

[0013] Furthermore, if the ranging device 2 is a LiDAR, the ranging device 2 emits light such as near-infrared or visible light in various directions, receives the reflected light reflected from various objects in the outside world, detects the time difference between the time of emission and the time of reception and the wavelength difference between the emitted light and the received light, and detects the distance and relative speed for each direction of the object in the outside world.

[0014] The imaging device 3 is an imaging device that includes an image sensor, such as a CMOS image sensor, and an imaging optical system. The imaging device 3 is mounted inside the passenger compartment of the vehicle 1, facing forward of the vehicle 1. In the example of FIG. 1 , the optical axis of the imaging device 3 is substantially aligned with a central axis 100 of the vehicle 1 at 0 degrees forward. The imaging device 3 captures an image of a field of view 103 in front of the vehicle 1 and outputs an imaging signal to the external environment recognition device. In this embodiment, the distance measuring device 2 and the imaging device 3 are configured to be installed separately, thereby ensuring a wide field of view.

[0015] FIG. 2 is a block diagram illustrating a hardware configuration of the external environment recognition device according to the first embodiment. The external environment recognition device 4 includes a computer equipped with an arithmetic unit 11 such as a central processing unit (CPU), a microprocessing unit (MPU), or a digital signal processor (DSP), a non-volatile memory 12 such as a read-only memory (ROM), a flash memory, or a hard disk drive, a volatile memory 13 called a random access memory (RAM), an input / output interface 14, and other peripheral circuits. These hardware components work together to run software and realize multiple functions. The external environment recognition device 4 may be composed of a single computer or multiple computers. The arithmetic unit 11 may be an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like.

[0016] The nonvolatile memory 12 stores programs capable of executing various calculations. That is, the nonvolatile memory 12 is a storage medium (storage device) from which the programs realizing the functions of this embodiment can be read. The volatile memory 13 is a storage medium (storage device) that temporarily stores the results of calculations performed by the calculation device 11 and signals input from the input / output interface 14. The calculation device 11 is a device that loads the programs stored in the nonvolatile memory 12 into the volatile memory and executes calculations, and performs predetermined calculations on data taken in from the input / output interface 14, the nonvolatile memory 12, and the volatile memory 13 in accordance with the programs.

[0017] The input / output interface 14 is connected to each of the distance measuring device 2 and the imaging device 3. The input section of the input / output interface 14 converts signals input from various devices (such as the distance measuring device 2 and the imaging device 3) into data that can be calculated by the arithmetic device 11. The output section of the input / output interface 14 generates an output signal according to the calculation result of the arithmetic device 11 and outputs the signal to the various devices.

[0018] 3 is a block diagram illustrating a functional configuration of the external environment recognition device according to the first embodiment. The external environment recognition device 4 includes a noise removal unit 41, an object detection unit 42, a geometric correction unit 43, a search range setting unit 44, and a model estimation unit 45.

[0019] The noise removal unit 41 removes what is considered noise from the distance information and relative speed information for each direction detected by the distance measuring device 2. Here, noise is considered to be, for example, distance information and relative speed information based on millimeter waves or light that are not directly reflected from an object. Alternatively, noise is considered to be distance information and relative speed information based on millimeter waves or light that are reflected by an object that is not intended to be a detection target. An object that is not intended to be a detection target is, for example, a stationary object that is always present, such as a road surface. For example, when the vehicle 1 is traveling, millimeter waves or light reflected from the road surface detects a relative speed equal to the traveling speed of the vehicle 1. When such a relative speed is detected, the noise removal unit 41 determines that the detected object is a road surface, taking into account information such as reflectivity, and removes the distance information and relative speed information related to that object.

[0020] The object detection unit 42 detects objects in the external world based on the distance information and relative speed information after the noise removal process by the noise removal unit 41. The object detection unit 42 detects a single object by combining distance information and relative speed information detected at multiple locations, taking into account information such as reflectivity. For example, if the distance measuring device 2 detects an object at a certain distance in a certain direction and then detects an object at approximately the same distance in a direction slightly offset from that direction, and if the relative speeds, reflectivities, etc. are similar, the two detected objects are considered to be the same object. Based on this concept, the object detection unit 42 detects actual objects from the distance information and relative speed information for each direction. The object detection unit 42 infers the center position of the object and uses the inferred center position as the reference position for the object. Note that the reference position may be a position other than the center position of the object, such as the object's road contact point. The object detection unit 42 sets the reference position of the detected object based on a vehicle coordinate system based on a predetermined position of the vehicle 1 (for example, the center position of the vehicle 1). If necessary, the object detection unit 42 converts the coordinate system based on the position of the distance measuring device 2 into a vehicle coordinate system based on the predetermined position of the vehicle 1.

[0021] The geometric correction unit 43 performs geometric correction on the image capture signal output by the imaging device 3 based on a geometric table pre-stored in the non-volatile memory 12. The geometric table includes camera parameters of the imaging device 3. The camera parameters include, for example, the focal length of the optical system of the imaging device 3, distortion information, and the installation position of the imaging device 3 in the vehicle coordinate system. The geometric correction unit 43 uses the geometric table to perform a well-known perspective transformation and output a corrected image in which the image capture signal has been geometrically corrected. The corrected image associates the two-dimensional coordinates of each pixel in the image capture signal captured by the imaging device 3 with the vehicle coordinate system. By using the corrected image, it is possible to determine the position of the object corresponding to each pixel in the vehicle coordinate system, provided that the depth of each pixel (i.e., the distance from the vehicle 1 to the object corresponding to each pixel) can be determined.

[0022] The search range setting unit 44 sets a partial range of the entire corrected image as a search range in which model estimation is performed by the model estimation unit 45. The search range setting unit 44 identifies the position of the object detected by the object detection unit 42 on the image signal geometrically corrected by the geometric correction unit 43, and sets a range including that position as a search range corresponding to the object.

[0023] The model estimation unit 45 performs model estimation for the corrected image using a model dictionary stored in advance in the non-volatile memory 12. The model dictionary includes type identification information for identifying the type of object. The model estimation unit 45 identifies whether the search range set by the search range setting unit 44 matches a model in the model dictionary, performs regression estimation of coordinates that match the model, and identifies the type of object and the coordinates of the object appearing within the search range. The model estimation unit 45 performs the identification and regression estimation based on a model dictionary such as weighting coefficients learned by machine learning. Note that, in this embodiment, it is assumed that the model dictionary is created in advance by machine learning, but the external environment recognition device 4 may dynamically create and update the model dictionary.

[0024] The result of the model estimation by the model estimation unit 45 is output from the external environment recognition device 4 to another device in the vehicle 1. For example, the result of the model estimation is output to another control device that predicts a collision between the vehicle 1 and another object. In such a control device, by using the result of the model estimation to predict a collision with the vehicle 1, it is possible to issue a warning to the driver of the vehicle 1 when a possibility of a collision between the vehicle 1 and an external object is predicted, or to perform vehicle control such as automatically applying the brakes to avoid a collision.

[0025] FIG. 4 is a flowchart of the external environment recognition process performed by the external environment recognition device 4 according to the first embodiment. In step S100, distance information and relative speed information are input from the distance measuring device 2 to the calculation device 11. In step S110, the noise removal unit 41 removes noise from the distance information and relative speed information input in step S100. In step S120, the object detection unit 42 detects an object using the distance information and relative speed information. In step S130, an image capture signal is input from the imaging device 3 to the calculation device 11. In step S140, the geometric correction unit 43 performs geometric correction on the image capture signal input in step S130 using a geometry table stored in the non-volatile memory 12, and outputs a corrected image. In step S150, the search range setting unit 44 executes a process of specifying the vertical angle of view and the horizontal angle of view, which will be described later. In step S160, the search range setting unit 44 sets a search range for the corrected image output in step S140 using the vertical angle of view and the horizontal angle of view specified in step S150. In step S170, the search range setting unit 44 stores information for additional learning in the non-volatile memory 12. The information for additional learning is, for example, information that serves as input (learning data) when updating the model dictionary through machine learning. Such machine learning may be performed outside the external environment recognition device 4 or by the computing device 11 while the external environment recognition device 4 is operating. Examples of the information to be stored include information indicating the position of the search range set in step S150 (i.e., information such as the height of the upper and lower ends, the positions of the left and right ends, the vertical angle of view, and the horizontal angle of view, as described below), a partial image extracted from the corrected image corresponding to the search range (i.e., brightness information and color information within the search range), and information regarding the position of objects within the search range (i.e., distance information, information indicating the height from the road surface, etc.). In step S180, the model estimation unit 45 performs model estimation for the corrected image based on the model dictionary and the search range set in step S160. In step S190, the model estimation unit 45 outputs the results of the model estimation performed in step S180 to the outside of the external environment recognition device 4.

[0026] A detailed description will be given of the process of specifying the vertical and horizontal angles of view by the search range setting unit 44. The search range setting unit 44 specifies the vertical and horizontal angles of view of the search range in the imaging device 3 in order to set a search range for model estimation.

[0027] FIG. 5 is an explanatory diagram illustrating a method for determining the vertical angle of view. In the following description, it is assumed that the model to be estimated by model estimation is a pedestrian. In FIG. 5, the pedestrian 50 to be estimated is located in front of the vehicle 1, a distance 53 away from the image capture device 3 in a direction directly ahead of the vehicle 1. The vertical distance of the image capture device 3 from the road surface is an installation height 54. When determining the vertical angle of view, the upper height 51 and the lower height 52 of the search range are determined in advance. While the size of the model to be estimated may vary from one individual to another, it is desirable that most individuals be included within the range from the upper height 51 to the lower height 52, and that this range not be too large. If many individuals are not included within this range, the accuracy of the model estimation will decrease. Furthermore, if the range is set too wide, the time required for model estimation may increase and the estimation accuracy may decrease.

[0028] The upper end height 51 is a height that is set as the upper limit when learning the model. For example, if the model is a pedestrian, it is considered that the majority of pedestrians can be covered by setting the upper end height 51 to about 180 cm to 250 cm. Similarly, for types of objects other than pedestrians, it is advisable to assume a height that is the upper limit of the height of the individual or a height that is several times the standard deviation from the average of the objects of the type to be modeled, which includes the upper ends of most individuals.

[0029] In model estimation, it is possible to identify the object type and locate the road contact point even if the entire object is not necessarily within the search range. Therefore, the upper end height 51 may be set to the average height in the area where the external environment recognition device 4 is operated plus several times the standard deviation of height in the area. If the model is very tall and does not fit within the upper end of the vertical angle of view of the imaging device 3, the height should be set to the height of the position seen at the upper end of the vertical angle of view of the imaging device 3 at the nearest distance to be recognized, similar to the lower end height 52 described below.

[0030] When an object to be recognized (e.g., a pedestrian, etc.) is present at the nearest distance that is desired to be recognizable, the height of the object is the higher of the lowest position that can be seen when it is blocked by the hood of the vehicle 1 or the position that appears at the bottom of the vertical angle of view of the image capture device 3. The reason why the contact point with the road surface is not simply set as the height of the bottom end 52 is that when an object is present near the image capture device 3, the bottom end of the object may be blocked by the hood of the vehicle 1 or may be outside the imaging range of the image capture device 3, in which case the object will not appear at the position corresponding to the contact point in the corrected image. The height of the bottom end 52 is determined based on the installation position of the image capture device 3, the imaging range of the image capture device 3, the model of the vehicle 1 (hood height), etc.

[0031] It is desirable to provide a margin for the upper end height 51 and the lower end height 52. In other words, it is desirable to set the upper end height 51 at a slightly higher position and the lower end height 52 at a slightly lower position. By doing so, even when the angle of view cannot be obtained as designed due to shaking of the imaging device 3 or manufacturing variations, a search range can be set in which the object to be modeled is reliably captured in the corrected image, thereby improving recognition accuracy.

[0032] In model estimation, it is determined whether an object captured in a search range set in the corrected image based on the position of an object detected by the distance measuring device 2 corresponds to a model, and it is also determined where in the image the object corresponding to the model is captured, including width and height that cannot be determined from distance information alone. The model dictionary used in model estimation is a dictionary of coefficients and the like of a model to be identified that has been created in advance using machine learning or the like. The search range setting unit 44 sets a search range according to the height 51 of the upper end and the height 52 of the lower end of the model assumed when determining the installation position of the imaging device 3 or when additional learning of the model to be identified is performed.

[0033] 6 is an explanatory diagram showing the vertical angle of view corresponding to the height 51 of the top end of the search range. First, a distance 55 to the top end height 51 is calculated using the height 54 of the imaging device 3 as a reference. Then, by applying a trigonometric function using the distance 53 from the imaging device 3 to the pedestrian 50 and the distance 55 to the top end height 51 based on the height 54 of the imaging device 3 as a reference, a vertical angle of view 56 to the top end height 51 can be calculated when an axis extending from the imaging device 3 parallel to the road surface is set to 0 degrees.

[0034] 7 is an explanatory diagram showing the vertical angle of view corresponding to the height 52 of the bottom end of the search range. First, a distance 57 to the bottom end height 52 is calculated using the height 54 of the imaging device 3 as a reference. Then, by applying a trigonometric function using the distance 53 from the imaging device 3 to the pedestrian 50 and the distance 57 to the bottom end height 52 based on the height 54 of the imaging device 3 as a reference, a vertical angle of view 58 to the bottom end height 52 is calculated when an axis extending from the imaging device 3 parallel to the road surface is set to 0 degrees. The vertical angle of view of the search range can be determined by adding the vertical angle of view 56 to the top end height 51 shown in FIG. 6 and the vertical angle of view 58 to the bottom end height 52 shown in FIG. 7 together.

[0035] FIG. 8 is an explanatory diagram illustrating a method for specifying the horizontal angle of view. When specifying the horizontal angle of view, a search range width 60 is determined in advance. Although individual differences in the size of the model to be estimated are expected, it is desirable that most individuals are included within the search range width 60 and that the range is not too large. If many individuals are not included within the range, the accuracy of model estimation will decrease. Furthermore, if the range is made excessively wide, the time required for model estimation may increase and the estimation accuracy may decrease.

[0036] The search range width 60 is set to be large enough to represent the width of an object that appears in a corrected image when performing model identification and regression estimation using model estimation. For example, if the model to be estimated is a pedestrian, the search range width 60 is set to about 100 cm. When a wide object is used as a model, the search range width 60 is set accordingly wide, and when a narrow object is used as a model, the search range width 60 is set accordingly narrow, thereby making it possible to reduce the search range for model estimation and the amount of calculations.

[0037] First, the right and left ends of the search range are identified relative to the center position of the object detected by object detection unit 42, using a predetermined search range width 60. Specifically, the right end of the search range is determined as a position shifted to the right by half the search range width 60 from the center position of the object, and the left end of the search range is determined as a position shifted to the left by half the search range width 60 from the center position of the object. Then, by applying trigonometric functions using the right and left ends of the search range and distance 53 from image capture device 3 to pedestrian 50, a horizontal angle of view 61 to the left end of the search range and a horizontal angle of view 62 to the right end of the search range can be determined when an axis extending from image capture device 3 in a direction directly in front of image capture device 3 is set to 0 degrees.

[0038] 9 is an explanatory diagram of a method for setting a search range in a corrected image. Fig. 9 shows an example of a corrected image, i.e., a perspective projection image 63 mapped to a perspective projection, and a search range 64 set on the image. Even for other projection planes, the vertical angle of view at the upper and lower ends and the horizontal angle of view at the left and right ends can be converted into image coordinates according to the formula for the height of the optical axis center.

[0039] The search range in the corrected image is set according to the following equations (1) to (4): where xleft is the horizontal image coordinate of the left end of the search range, xright is the horizontal image coordinate of the right end of the search range, ytop is the vertical image coordinate of the top end of the search range, ybottom is the vertical image coordinate of the bottom end of the search range, f is the focal length of the imaging unit of the imaging device, ph is the horizontal pixel pitch of the image sensor, pv is the vertical pixel pitch of the image sensor, θleft is the horizontal angle of view corresponding to the distance to the left end of the search range (corresponding to horizontal angle of view 61 in FIG. 8 ), θright is the horizontal angle of view corresponding to the distance to the right end of the search range (corresponding to horizontal angle of view 62 in FIG. 8 ), θ1 is the vertical angle of view corresponding to the height of the top end of the search range (corresponding to vertical angle of view 56 in FIG. 6 ), θ2 is the vertical angle of view corresponding to the height of the bottom end of the search range (corresponding to vertical angle of view 58 in FIG. 7 ), wimage is the image width, and himage is the image height.

[0040]

[0041] Note that variations occur between the calculated object position and its position on the image due to variations in the object being the subject of model estimation, the attitude of the imaging device 3, distance measurement errors by the distance measuring device 2, etc. Therefore, it is desirable to provide some margin in the width and height of the left and right ends and top and bottom ends of the search range calculated by the above formulas (1) to (4).

[0042] When performing regression estimation (described later), if the top, bottom, left, or right edges are not included in the search range, it will hinder accurate regression estimation, so it is desirable to set larger margins for width and height. Also, when classifying by height (described later), in order to accurately estimate the cut-out parts of the pedestrian's body that appear in the search range, it is better to expand the width and height based on the physical width and height rather than the number of pixels on the image. This will ensure that the body parts that appear in the search range for a specific height do not change regardless of distance, enabling accurate height estimation.

[0043] FIG. 10 is an explanatory diagram showing an example of a model configuration when model estimation is performed using a binary classifier. Below, an example is shown in which pedestrian characteristics are defined as a model and a binary classification of whether or not an object is a pedestrian is performed. In this case, machine learning is performed on a large number of images in which pedestrian portions have been cut out in advance, and pedestrian models are entered as coefficients in a model dictionary. The external environment recognition device 4 uses this model dictionary to classify pedestrians from corrected images. While a binary example is used in this example, objects other than pedestrians, such as vehicles and cyclists, can also be classified by defining a multi-class model with three or more values.

[0044] For example, as shown in image 71 and image 72 in FIG. 10, a large number of images containing pedestrians are prepared, which are captured in the external environment where the external environment recognition device 4 is actually used. Then, images in which pedestrians are cut out from these images are used as a pedestrian class for machine learning. Similarly, as shown in image 73 in FIG. 10, a large number of images are prepared that show the background of a scene in which the pedestrian may be present. These images are used as a background class for machine learning.

[0045] Using the pedestrian class images (e.g., image 71 and image 72) and background class images (e.g., image 73) prepared in this manner as input, machine learning is performed using, for example, a convolutional neural network (CNN), and co-occurring image features of the images used for learning are entered as coefficients into a model dictionary. If the features described in the model dictionary are similar (co-occur) to the image appearing in the search range of model estimation in the corrected image, the corresponding model is output as the identification result. For example, if they co-occur with the pedestrian class, the search range of the image is identified as containing a pedestrian, and if they do not co-occur, the search range of the image is identified as containing an object other than a pedestrian. By performing machine learning on various possible variations to create an identification model dictionary, object types can be identified with a high identification rate.

[0046] FIG. 11 is an explanatory diagram showing an example of a model configuration for performing regression estimation. Below, an example is shown in which the characteristics of a pedestrian are defined as a model and regression estimation of a pedestrian is performed. In regression estimation, the coordinates of a specific position (e.g., left, right, top, or bottom) of a modeled object within a search range are estimated. Images containing the model (e.g., images 74, 75, and 76 shown in FIG. 11 ) and the coordinates of a specific position of the model (e.g., left, right, top, or bottom, e.g., images 78, 79, and 790) are provided as training data, and machine learning is performed using, for example, a CNN. At this time, an image that does not contain the model to be estimated (e.g., image 77 shown in FIG. 11 ) is also input as training data.

[0047] FIG. 12 is an explanatory diagram showing an example of the configuration of a model for solving a multivalue classification problem, in which different classes are defined based on height. For example, the height of the top end of a search range is set to 200 cm and the height of the bottom end to 50 cm, and images of pedestrians captured within that range are cut out and trained into classes based on height. FIG. 12 shows training images 80 of pedestrians with heights of approximately 200 cm, 81 of pedestrians with heights of approximately 180 cm, 82 of pedestrians with heights of approximately 160 cm, and 83 of pedestrians with heights of approximately 140 cm. In this way, a large number of training images are prepared, each divided into four classes from 200 cm to 140 cm, in 20 cm increments, and trained as different classes.

[0048] By dividing the classes into height classes in this way, it is possible to estimate not only the type of object but also its height and body height at the same time. In this example, the classes are divided every 20 cm, with a maximum of 200 cm and a minimum of 140 cm, but it is also possible to divide into more classes to increase the resolution and the range of heights that can be distinguished.

[0049] 13 is an explanatory diagram illustrating an example of information stored as learning data for updating the model dictionary by the search range setting unit 44. As described in step S170 of Fig. 4, the search range setting unit 44 stores, for example, a partial image obtained by cutting out a portion corresponding to the search range from the corrected image (i.e., the portion showing the model), information indicating where in the corrected image the heights of the upper and lower ends of the search range correspond, distance information, height information from the road surface, and the like.

[0050] For example, as shown in Fig. 13, assume that the imaging device 3 captures an image 84. At this time, the search range setting unit 44 sets the area containing a pedestrian, who is the model of the estimation target, as the search range. The search range setting unit 44 cuts out and stores a partial image 85 corresponding to the search range from the image 84. Depending on the number of pixels of the image sensor of the imaging device 3, the data size of the image 84 may become large. Therefore, it is desirable to reduce the data size of the image for additional learning and save storage space by cutting out an image near an object whose model is to be defined based on information about the detected object, as in the partial image 85.

[0051] The search range setting unit 44 stores the partial image 85 together with the distance information of the pedestrian included therein, which was acquired by the distance measuring device 2. The distance information 86 illustrated in FIG. 13 is an illustration of the results of distance measurement by the distance measuring device 2 arranged as in image 84. What is actually stored is partial distance information 87, which is obtained by cutting out an area corresponding to the search range from the distance information 86, just like the partial image 85 for image 84. As with the case of storing the partial image 85, storing the partial distance information 87 instead of the distance information 86 can save storage space.

[0052] The search range setting unit 44 stores height information of objects appearing in the partial image 85 along with the partial distance information 87. The height information 88 illustrated in FIG. 13 is height information calculated for each vertical and horizontal angle of view in the image based on the image captured by the imaging device 3, the distance information acquired by the distance measuring device 2, and the information of the object detected by the object detection unit 42. As with the partial image 85 for the image 84, what is actually stored is partial height information 89, which is obtained by cutting out an area corresponding to the search range from the height information 88. As with the case of storing the partial image 85, storing the partial height information 89 instead of the height information 88 can save storage space.

[0053] In addition to the various types of information described above, the results of model estimation by the model estimation unit 45 may be stored for additional learning. In this case, step S170 in the flowchart shown in FIG. 4 needs to be performed after the model estimation process in step S180.

[0054] In this way, when the heights of the upper and lower ends of the search range are set according to the specifications of the vehicle 1 and the specifications and installation position of the imaging device 3, by storing the image, distance information, height information, etc. for which a model is to be defined as shown in Fig. 13, it becomes possible to freely extract and learn learning images for machine learning. This eliminates the need to collect learning images, which is an enormously time-consuming task, for each vehicle, and reduces the development effort.

[0055] According to the above-described first embodiment, the following advantageous effects are achieved.

[0056] (1) The computing device 11 detects an object using the distance measurement results from the distance measurement device 2 (distance measurement unit), sets a search range in which model estimation processing is performed on an image captured by the imaging device 3 (imaging unit) using the object detection results, and performs model estimation of the object in the search range from an image captured in a range from a height (first height) at the top end of the search range that is higher than the ground point of the object to a height (second height) at the bottom end of the search range using type identification information (model dictionary) for identifying the type of object stored in the non-volatile memory 12 (predetermined storage unit).In this way, it is possible to reliably recognize objects located near the imaging device (camera) 3, for example, without a part of the object deviating from the field of view of the imaging device 3 or a part of the object being obstructed by an obstruction.

[0057] (2) The computing device 11 uses a discrimination model for discriminating the type of object, which makes it possible to discriminate nearby objects.

[0058] (3) The calculation device 11 uses a regression model that performs regression estimation of object position information, making it possible to estimate detailed position information of nearby objects.

[0059] (4) The computing device 11 uses a classification model that classifies objects by their height. This allows the computing device 11 to simultaneously estimate not only the type of object but also its height (body height) even when the area around the target object is not visible.

[0060] (5) The calculation device 11 sets the height of the top end of the search range (first height) according to the height of the top end of the object defined in the model estimation. This allows the model to occupy a larger proportion of the image range of the search range for model estimation, thereby reducing data volume.

[0061] (6) The computing device 11 sets the height of the bottom end of the search range (second height) according to the height of the bottom end of a nearby object that appears in the image without being occluded by the hood of the vehicle 1. This makes it possible to perform model estimation in the vicinity by not using the features of the occluded part, while increasing the proportion of the model in the image range of the search range for model estimation, thereby saving data volume.

[0062] (7) The computing device 11 stores an image of the detected object and distance information corresponding to the image. This allows for model estimation in the vicinity without using the features of the occluded part, and also increases the proportion of the model in the image range of the search range for model estimation, thereby saving data capacity.

[0063] <Second embodiment> An external environment recognition device according to a second embodiment of the present invention will be described with reference to Figures 14 to 16. Note that the same reference symbols are used for configurations that are the same as or equivalent to those described in the first embodiment, and differences will be mainly described.

[0064] FIG. 14 is a diagram similar to FIG. 1 and is a schematic diagram of a vehicle equipped with an external environment recognition device according to a second embodiment. The vehicle 1 is equipped with an imaging device 5 instead of the distance measuring device 2 and the imaging device 3. The imaging device 5 is a so-called stereo camera, and incorporates an imaging unit 5a provided on the left side and an imaging unit 5b provided on the right side. The imaging unit 5a captures an image of a field of view 201 on the front left of the vehicle 1 and outputs an imaging signal to the external environment recognition device. The imaging unit 5b captures an image of a field of view 202 on the front right of the vehicle 1 and outputs an imaging signal to the external environment recognition device. The imaging units 5a and 5b are installed a known distance apart. This distance may be fixed or variable. The external environment recognition device of this embodiment is configured to be able to grasp this distance.

[0065] Since there is a parallax between the image capturing units 5a and 5b, as is well known, distance measurement can be performed by utilizing this parallax. In other words, the image capturing device 5 can output image signals by capturing images and acquire distance information by distance measurement using the image capturing units 5a and 5b. The area where the field of view 201 and the field of view 202 overlap is the stereo field of view, and is the area where distance measurement can be performed by utilizing parallax.

[0066] 15 is a block diagram similar to FIG. 3, which schematically shows the functional configuration of the external environment recognition device according to the second embodiment. The external environment recognition device 204 includes an object detection unit 242, a geometric correction unit 243, and a disparity calculation unit 246 instead of the noise removal unit 41, the object detection unit 42, and the geometric correction unit 43.

[0067] The geometric correction unit 243, like the geometric correction unit 43 according to the first embodiment, performs geometric correction processing on the imaging signals output from the imaging units 5a and 5b. The geometric table according to this embodiment is a parallelization geometric table in which epipolar lines are aligned horizontally so that the image of one imaging unit 5a and the image of the other imaging unit 5b show the same object shifted horizontally by a parallax corresponding to the depth distance relative to the image of the other imaging unit 5a, making it convenient for the parallax calculation unit 246 to perform parallax calculation. This geometric table is created, for example, in accordance with a perspective projection plane perpendicular to the parallel optical axes of the base points of the two imaging units 5a and 5b.

[0068] The parallax calculation unit 246 calculates the parallax of the same object using a pair of corrected images that have been geometrically corrected by the geometric correction unit 243. Specifically, it performs a so-called stereo matching process to determine which pixel in the corrected image of one imaging unit 5 a corresponds to which pixel in the corrected image of the other imaging unit 5 b (whether they correspond to the same object), and outputs a parallax image in which the parallax for each pixel is arranged. For stereo matching, a matching method such as SAD (Sum of Absolute Differences) can be used.

[0069] The object detection unit 242 detects objects that appear in the disparity images and the corrected images. For example, it detects three-dimensional objects by clustering objects at the same distance using the U-Disparity images.

[0070] 16 is a flowchart of the external environment recognition process by the external environment recognition device 204 according to the second embodiment, similar to FIG. 4. In the flowchart of FIG. 16, steps S200 to S230 are executed instead of steps S100 to S140 in the flowchart of FIG.

[0071] In step S200, a pair of imaging signals is input from the imaging device 5 to the arithmetic unit 11. In step S210, the geometric correction unit 243 performs geometric correction on the pair of imaging signals input in step S200 using a geometric table stored in the nonvolatile memory 12, and outputs a pair of corrected images. In step S220, the parallax calculation unit 246 performs parallax calculation on the pair of corrected images output in step S210 using the geometric table stored in the nonvolatile memory 12, and outputs a parallax image. In step S230, the object detection unit 242 detects an object using the pair of corrected images and the parallax image (which corresponds to distance information since it has parallax according to distance).

[0072] Since the imaging device 5 is attached to the vehicle 1, which is a moving body, and captures images while moving, the images taken by the imaging units 5a and 5b must be captured almost simultaneously. If there is a time difference between the images captured by the imaging units 5a and 5b, the object captured in the image will move as the vehicle 1 moves, and the error included in the parallax value required for distance measurement will increase.

[0073] According to the second embodiment described above, the following advantageous effects are achieved.

[0074] (1) The external environment recognition device 204 calculates the distance to an object using the imaging device 5, which is a stereo camera. This configuration eliminates the need to separately install an imaging device and a distance measuring device, thereby reducing restrictions on the design of the vehicle 1. Furthermore, the number of parts is reduced and there is no need to separately align the distance measuring device, which reduces the assembly cost of the vehicle 1.

[0075] The following modified examples are also within the scope of the present invention, and it is possible to combine the configuration shown in the modified example with the configuration described in the above embodiment, to combine the configurations described in the different embodiments above, or to combine the configurations described in the different modified examples below.

[0076] <Variation 1> The number of distance measuring devices 2 and imaging devices 3 may be different from the numbers shown in FIG. 1 . For example, two or more imaging devices 3 may be installed. Also, only one distance measuring device 2 may be installed, or three or more distance measuring devices 2 may be installed. When multiple distance measuring devices 2 are present, the search range setting unit 44 may set a search range for each distance measuring device 2. In this case, multiple search ranges that are considered to correspond to the same object may be integrated into one search range to reduce the number of times model estimation is performed. In this way, the delay until the model estimation result is output can be reduced.

[0077] <Modification 2> The ranging device 2 and the imaging device 3 may be built into the external environment recognition device 4. In other words, functional units corresponding to the ranging device 2 and the imaging device 3 may be provided inside the external environment recognition device 4, rather than being provided outside the external environment recognition device 4. The same applies to the imaging device 5 according to the second embodiment. In this way, the functional units corresponding to the ranging device 2 and the imaging device 3 can be connected to the arithmetic device 11 via a faster, lower-delay interconnect, and the latency that occurs in the transfer of imaging signals and distance information can be reduced.

[0078] <Variation 3> Data other than the model dictionary may be used for model estimation. For example, weighting coefficients of a deep neural network (DNN) may be stored in the nonvolatile memory 12, and model estimation may be performed using a so-called deep learning technique. Also, model templates may be stored, and model estimation may be performed using a so-called template matching technique.

[0079] Although the embodiments of the present invention have been described above, the above embodiments merely illustrate some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above embodiments.

[0080] 1...vehicle, 2, 2a, 2b...range measuring device, 3, 5...imaging device, 4, 204...external environment recognition device, 5a, 5b...imaging unit, 11...arithmetic unit, 12...non-volatile memory, 13...volatile memory, 14...input / output interface, 41...noise removal unit, 42, 242...object detection unit, 43, 243...geometric correction unit, 44...search range setting unit, 45...model estimation unit, 246...parallax calculation unit

Claims

1. An on-board external environment recognition device equipped with a computing device and mounted on a vehicle having an imaging unit and a ranging unit, wherein the computing device detects an object using a ranging result by the ranging unit, uses the object detection result to set a search range in which model estimation processing is performed on an image captured by the imaging unit, and performs model estimation of the object in the search range from an image captured in a range from a first height to a second height higher than the ground point of the object, using type identification information for identifying the type of the object stored in a specified memory unit.

2. An in-vehicle external environment recognition device according to claim 1, wherein the computing device uses an identification model for identifying the type of the object.

3. An in-vehicle external environment recognition device according to claim 1, wherein the computing device uses a regression model for performing regression estimation of the object's position information.

4. An in-vehicle external environment recognition device according to claim 1, wherein the computing device uses an identification model that identifies classes of the object according to their heights.

5. An in-vehicle external environment recognition device according to claim 1, wherein the computing device sets a first height according to the height of the upper end of the object defined by the model estimation.

6. An in-vehicle external environment recognition device according to claim 1, wherein the computing device sets the second height according to the height of a lower end of a nearby object that is not blocked by the vehicle hood and appears in the image.

7. An in-vehicle external environment recognition device according to claim 1, wherein the computing device stores an image of a detected object and distance information corresponding to the image.

Images