Environmental recognition device and environmental recognition method

Abstract

An environment recognition device according to the present invention includes: a three-dimensional object detection unit that sets a detection region for detecting an object to be identified with respect to an image captured by an image-capturing unit, and detects a three-dimensional object captured in the image on the basis of the detection region; a three-dimensional object size calculation unit that calculates the size of the three-dimensional object detected in the detection region using information regarding a distance from an own device to the three-dimensional object that is calculated from the image; a three-dimensional object identification unit that identifies that the three-dimensional object detected in the detection region is the object to be identified; and a lateral speed calculation unit that limits a sampling range of the three-dimensional object on the basis of the size of the three-dimensional object, and calculates movement information in which the three-dimensional object to be identified moves in a lateral direction on the basis of information regarding an edge of the three-dimensional object to be identified.

Description

Technical Field

[0005]

[0001] The present invention relates to an environment recognition device and an environment recognition method.

Background Art

[0002] In recent years, with the spread of in-vehicle sensing devices, there has been a demand for improving the performance of collision safety functions for pedestrians. As an evaluation of the safety of automobiles (automobile assessment), a New Car Assessment Programme (NCAP) has been established. For this reason, Euro NCAP, Japan NCAP, etc. are used in each region. All evaluations for automobile assessment are being actively added or changed. Since the New Car Assessment Programme continues to evolve, it is necessary to keep up with the improvement of the performance of collision safety functions. As an evaluation of the performance of collision safety functions, one is the evaluation of the automatic braking of a pedestrian who suddenly jumps out.

[0003] In order for an ECU mounted on a vehicle to determine a pedestrian's sudden jump, first, a process of identifying a pedestrian from an image captured by an in-vehicle camera and calculating the lateral movement speed of the pedestrian is performed. Next, when the ECU determines that the pedestrian is moving in the direction of the host vehicle lane based on the movement speed and movement direction of the pedestrian, it performs an Autonomous Emergency Braking (AEB). ​​​​​​​​​​​​​​ Japanese Patent Publication No. 2012-203884 [Overview of the project] [Problems that the invention aims to solve]

[0006] Traditionally, 3D object recognition processing performed by the ECU (Electronic Control Unit) assumed that the 3D object was accurately detected, and reduced processing costs were achieved by maintaining recognition performance when identifying the detected area of ​​the 3D object. However, in reality, 3D objects may not be accurately detected at night, due to the color of clothing and background, distance, etc. For example, if a utility pole or signpost is near a pedestrian, the detection area may be larger than the actual size of the pedestrian.

[0007] One method for calculating the lateral velocity of a pedestrian uses the edges of the pedestrian within the detection area. In the technology described in Patent Document 1, edges (contours) are extracted from an image, and based on the size of the edges, it is determined whether the three-dimensional object detected in the detection area is a pedestrian. If the determination result is a pedestrian, the pedestrian's lateral velocity is calculated; otherwise, it is treated as a non-pedestrian. However, in the technology described in Patent Document 1, accurate edges cannot be extracted depending on the background conditions of the pedestrian. As a result, there was a risk that the edge would be treated as a non-pedestrian, and the accurate lateral velocity would not be calculated. Furthermore, in the method using edges within the detection area, as mentioned above, if an area larger than the actual size of the pedestrian is detected, an incorrect lateral velocity may be calculated for a stationary pedestrian. When such an incorrect lateral velocity is calculated, the ECU's vehicle control processing has problems such as issuing warnings or applying sudden brake control even in situations where there is no risk of a pedestrian suddenly stepping out.

[0008] This invention was made in view of these circumstances, and aims to accurately calculate the movement information of a three-dimensional object that is to be identified. [Means for solving the problem]

[0009] The environmental recognition device according to the present invention includes a three-dimensional object detection unit that sets a detection area for detecting an object to be identified in an image captured by an imaging unit, and detects three-dimensional objects in the image based on the detection area, and a three-dimensional object size calculation unit that calculates the size of the three-dimensional object detected in the detection area using distance information from the device to the three-dimensional object calculated from the image. Based on images from multiple consecutive frames, A three-dimensional object identification unit identifies that a three-dimensional object detected in the detection area is the object to be identified, and limits the sampling range of the three-dimensional object based on its size, and identifies the edge information of the three-dimensional object that is the object to be identified. As a result, the average value of the horizontal coordinates of points sampled from the edges of a three-dimensional object is calculated for each frame, and the edge information is obtained. Based on this, the three-dimensional object being identified moves laterally. Speed Movement information as It comprises a movement information calculation unit that calculates movement information. [Effects of the Invention]

[0010] According to the present invention, by limiting the sampling range of a three-dimensional object based on its size, it is possible to accurately calculate the movement information of the object to be identified, even when the detection of the three-dimensional object is unstable. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing an example of the overall configuration of an in-vehicle stereo camera device according to the first embodiment of the present invention. [Figure 2] This flowchart and configuration diagram show examples of processing performed in each functional unit of an in-vehicle stereo camera device according to the first embodiment of the present invention. [Figure 3] This flowchart shows an example of the process for calculating the lateral velocity of a three-dimensional object, which forms the basis of the present invention. [Figure 4] This figure shows the results of the 3D object detection process based on the camera image according to the first embodiment of the present invention. [Figure 5A] This figure shows an example of the right edge of a pedestrian captured within the detection area. [Figure 5B] This figure shows an example of the average horizontal position of the right edge calculated from the t-th frame image. [Figure 5C]It is a diagram showing an example of the horizontal position average of the right edge calculated from the image of the (t + 1) - th frame. [Figure 6A] It is a diagram showing an example of a detection result when there is an object outside the target near a pedestrian. [Figure 6B] It is a diagram showing an example of the detection result of the second frame. [Figure 6C] It is a diagram showing an example of the amount of movement of the horizontal position average. [Figure 7A] It is an example of a detection area including a pedestrian. [Figure 7B] It is an enlarged view of the right side part of the pedestrian photographed in the second frame. [Figure 8A] It is a diagram showing an example of a detection area including a pedestrian. [Figure 8B] It is a diagram showing an example where a detection area larger than the target is acquired. [Figure 9] It is a flowchart showing an example of the detailed processing of the three - dimensional object identification processing according to the first embodiment of the present invention. [Figure 10] It is a block diagram showing an example of the hardware configuration of a computer according to the first embodiment of the present invention. [Figure 11] It is a block diagram showing an example of the configuration of an in - vehicle camera device according to the second embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments for carrying out the present invention will be described with reference to the accompanying drawings. In this specification and the drawings, for components having substantially the same function or configuration, the same reference numerals are given and redundant explanations are omitted. The present invention is applicable to, for example, an in - vehicle ECU (Electronic Control Unit) for vehicle control that can communicate with an advanced driver assistance system (ADAS: Advanced Driver Assistance System) or an arithmetic device for autonomous driving (AD: Autonomous Driving).

[0013] [First Embodiment] Figure 1 is a block diagram showing an example of the overall configuration of an in-vehicle stereo camera device 100 according to the first embodiment of the present invention.

[0014] The on-board stereo camera device 100 is an example of an environmental recognition device mounted on a vehicle (not shown) that recognizes the external environment based on image information captured with the area in front of the vehicle, which is the direction of travel, as the target area. Based on the image information, the on-board stereo camera device 100 identifies, for example, road markings, pedestrians, vehicles, other three-dimensional objects, traffic lights, signs, and illuminated lamps. Furthermore, based on the results of identifying the external environment, the on-board stereo camera device 100 adjusts the brakes, steering, and other functions of the vehicle equipped with the on-board stereo camera device 100 (hereinafter referred to as "the vehicle"). The on-board stereo camera device 100 safely controls the vehicle by calculating the movement information (speed, distance, or direction of movement) of pedestrians moving laterally. In the following description, the direction in which a pedestrian moves across the front of the vehicle on which the device (on-board stereo camera device 100) is mounted, or across the vehicle's lane, is referred to as the "lateral direction." The lateral direction may be perpendicular to the direction of travel of the vehicle or the direction of the lane, or it may be diagonal.

[0015] The in-vehicle stereo camera device 100 includes a left camera 101, a right camera 102, an image input I / F (Interface) 103, an image processing unit 104, an arithmetic processing unit 105, a storage unit 106, a CAN (Controller Area Network) IF 107, a control processing unit 108, and an internal bus 109.

[0016] The left camera 101 and the right camera 102 are positioned on the left and right sides of the vehicle, respectively. Each camera has an image sensor that captures the environment in front of the vehicle and acquires image information. In the following description, the left camera 101 and the right camera 102 will also be referred to as the "stereo camera" and the "imaging unit." The image information output by the image sensor of the stereo camera will be abbreviated as "image." The imaging unit (left camera 101, right camera 102) is a stereo camera having two image sensors. In Figure 1, the stereo camera is composed of two left cameras 101 and right cameras 102, with each camera having one image sensor. However, even if a single camera has two image sensors arranged horizontally, that camera may be treated as a stereo camera.

[0017] The image input IF 103 is connected to the image sensor of the stereo camera. The image input IF 103 controls the imaging operation of the stereo camera and captures the images captured by the stereo camera. The image data captured through the image input IF 103 is sent to each unit via the internal bus 109. The image processing unit 104 and the arithmetic processing unit 105 then perform predetermined processing on the image data, and the intermediate results and the final image data are stored in the storage unit 106.

[0018] The image processing unit 104 compares the first image obtained from the image sensor of the left camera 101 with the second image obtained from the image sensor of the right camera 102, and performs image correction on each image, such as correcting for device-specific deviations caused by the image sensor and noise interpolation. The image processing unit 104 also stores the corrected image in the storage unit 106. Furthermore, the image processing unit (image processing unit 104) calculates the parallax of a three-dimensional object based on the first and second images input from the left and right image sensors, respectively. In this process, the image processing unit 104 calculates the corresponding locations between the first and second images and calculates the parallax information. Subsequently, the image processing unit 104 stores the parallax information in the storage unit 106, similar to the process used to store the corrected image in the storage unit 106. In the following description, the first image and the second image will be abbreviated as "image" if they are not distinguished.

[0019] The arithmetic processing unit 105 uses the image stored in the memory unit 106 (image captured by the left camera 101 or the right camera 102) and the parallax information (distance information between each point on the image) to perform identification processing of various three-dimensional objects necessary for identifying the environment around the vehicle. Examples of these three-dimensional objects include people, cars, other obstacles, traffic lights, signs, car taillights or headlights, etc. The identification results and some of the intermediate calculation results from the arithmetic processing unit 105 are stored in the memory unit 106. The control processing unit 108 uses the image identification results stored in the memory unit 106 to calculate the control of the vehicle.

[0020] The control processing unit 108 calculates a vehicle control policy using the image and the identification results of various three-dimensional objects obtained by the arithmetic processing unit 105. The vehicle control policy obtained as a result of the calculation by the control processing unit 108, and some of the identification results of various three-dimensional objects by the arithmetic processing unit 105, are transmitted to the in-vehicle network CAN 110 via CAN IF 107, and the vehicle is braked by an accelerator or brake (not shown). Furthermore, the control processing unit 108 monitors these operations to ensure that each processing unit is not malfunctioning and that no errors have occurred during data transfer, thus preventing abnormal operation.

[0021] The image processing unit 104 is connected to the image input IF 103, the arithmetic processing unit 105, the storage unit 106, and the control processing unit 108 via the internal bus 109. The image processing unit 104 is also connected to the CAN IF 107 of the in-vehicle network CAN 110, which is located outside the in-vehicle stereo camera device 100, via the internal bus 109. As will be described later, the control processing unit 108, the image processing unit 104, the arithmetic processing unit 105, the storage unit 106, and the CAN IF 107 are composed of one or more computer units.

[0022] The storage unit 106 is composed of, for example, a memory that stores images processed by the image processing unit 104, scan results obtained by scanning images by the arithmetic processing unit 105, and so on. The CAN IF107 handles input and output to an external in-vehicle network. Therefore, the CAN IF107 outputs information output from the in-vehicle stereo camera device 100 to the vehicle's control system (not shown) via the in-vehicle network CAN110.

[0023] Figure 2 is a flowchart and configuration diagram showing examples of processes performed by each functional unit of the in-vehicle stereo camera device 100. Figure 2 shows an image recognition process as an example of an environment recognition method performed by the in-vehicle stereo camera device 100. Each process shown in Figure 2 is described by corresponding it to the reference numerals of the functional units shown in Figure 1.

[0024] First, images are captured by the stereo cameras (left camera 101, right camera 102) of the in-vehicle stereo camera device 100. The image processing unit 104 performs image processing (S1), such as correction to absorb the inherent characteristics of the stereo camera's image sensor, on both the image data 201 captured by the left camera 101 and the image data 202 captured by the right camera 102. The results of the image processing are stored in the image data buffer 203. The image data buffer 203 is provided in the storage unit 106 in Figure 1 and has the function of temporarily storing image-related data.

[0025] Next, the image processing unit 104 compares the two corrected images read from the image data buffer 203. At this time, the image processing unit 104 performs parallax processing (S2) to obtain parallax information of the images output by the stereo camera's image sensor (referred to as "left and right images") by comparing the two images. The parallax information of the left and right images reveals where a certain point of interest on a three-dimensional object corresponds to which points on the stereo camera's image, and the distance to the object can be obtained using the principle of triangulation.

[0026] As described above, image processing (S1) and parallax processing (S2) are performed in the image processing unit 104 in Figure 1, and the final processed image, parallax information, and distance to the object are stored in the storage unit 106.

[0027] Next, the 3D object detection unit 301 of the arithmetic processing unit 105 uses the parallax information obtained in the parallax processing (S2) to perform 3D object detection processing (S3) for detecting 3D objects in 3D space. In 3D object detection processing (S3), the 3D object detection unit (301) performs processing to detect 3D objects that are the same distance from the device and the vehicle on which the stereo camera is mounted. As shown in Figure 3 described later, 3D object detection processing (S33) includes image acquisition processing (S31) and 3D object detection processing (S32) by the 3D object detection unit 301, as well as processing to calculate the size of the 3D object by the 3D object size calculation unit 302.

[0028] In the 3D object detection process (S3), the 3D object detection unit (301) acquires images captured by the imaging unit (left camera 101, right camera 102), sets a detection area for detecting the object to be identified in this image, and detects 3D objects in the image based on the detection area (S31, S32 in Figure 3). The 3D object size calculation unit (302) calculates the size of the 3D object detected in the detection area using distance information from the device to the 3D object calculated from the image (S33 in Figure 3).

[0029] Next, the 3D object recognition unit 303 of the arithmetic processing unit 105 performs 3D object recognition processing (S4) using the image, parallax information, and distance to the object stored in the storage unit 106. The 3D objects to be recognized in 3D object recognition processing (S4) include people, cars, other 3D objects, signs, traffic lights, taillights, etc. The details of 3D object recognition processing (S4) are determined by the characteristics of the object and constraints such as the processing time available to the system. 3D object recognition processing (S4) includes 3D object recognition processing (S41) by the 3D object recognition unit 303, as shown in Figure 3, and lateral speed calculation processing (S42) by the lateral speed calculation unit 304.

[0030] In the 3D object recognition process (S4), the 3D object recognition unit (303) identifies that the 3D object detected in the detection area is the object to be identified (S41). 3D objects identified as objects to be identified by the 3D object recognition unit (303) are those that are at the same distance in two images. The 3D object recognition unit (303) identifies that a 3D object is the object to be identified based on images from multiple consecutive frames. The 3D object recognition unit 303 can identify a pedestrian as the object to be identified by matching the posture of a person walking or the shape of a person.

[0031] The movement information calculation unit (lateral velocity calculation unit 304) limits the sampling range of the object (pedestrian) based on the size of the object, and calculates movement information (lateral velocity) of the object moving laterally based on the edge information of the object to be identified (S42). Here, the movement information calculation unit (lateral velocity calculation unit 304) calculates the average value of the horizontal coordinates of the points from which the edges of the object have been sampled for each frame as edge information, and calculates the lateral velocity of the object to be identified as movement information.

[0032] The movement information calculation unit (lateral speed calculation unit 304) samples either the left or right edge of the object (pedestrian) and calculates the average value of the horizontal coordinates of the sampled edge. The lateral speed calculation unit 304 samples edges from the object, for example, from the part closer to the road surface (e.g., toes, heels) to the part further from the road surface (e.g., head, hat). If the object being identified is a motorcycle, the edges sampled will be the motorcycle's wheels closer to the road surface and the helmet further from the road surface. In countries with left-hand traffic, the right edge of the pedestrian is sampled, and in countries with right-hand traffic, the left edge of the pedestrian is sampled. However, the lateral speed calculation unit 304 may arbitrarily change the sampled edge to the right or left depending on the direction in which the object being identified (pedestrian) is expected to jump into the vehicle's lane.

[0033] Next, the control processing unit 108 performs vehicle control processing (S5) taking into account the identification result of the three-dimensional object and the state of the vehicle (speed, steering angle, etc.). Through this vehicle control processing (S5), the control processing unit 108, for example, issues a warning to the vehicle occupants and performs braking such as braking and steering angle adjustment of the vehicle. The control processing unit 108 also decides on a policy to perform avoidance control of the object by braking and outputs the result to the in-vehicle network CAN110 via CAN IF107.

[0034] The 3D object recognition process (S4) shown in Figure 2 is performed by the arithmetic processing unit 105 in Figure 1, and the vehicle control process (S5) is performed by the control processing unit 108 in Figure 1. Output to the in-vehicle network CAN 110 is performed by CAN IF 107. Each of these processes and each functional unit is composed of, for example, one or more computer units and is configured to exchange data with each other.

[0035] Furthermore, the parallax processing (S2) performed in the image processing unit 104 obtains the parallax or distance at each pixel of the left and right images. Then, in the 3D object detection processing (S3) performed in the 3D object detection unit 301, pixels with the same parallax or distance are grouped together as 3D objects in 3D space. Note that the 3D object detection processing (S3) does not detect the type of 3D object, so if a person is standing near a tree, the tree and the person may be grouped together. Also, if multiple people are standing at the same distance, multiple people may be grouped together. After the 3D object detection processing (S3), 3D object identification processing (S4) is performed based on the position on the image and the detection area for detecting 3D objects.

[0036] Figure 3 is a flowchart showing an example of the process for calculating the lateral velocity of a three-dimensional object, which forms the basis of the present invention. Here, we will explain the process focusing on the three-dimensional object detection process (S3), various identification processes (S4), and vehicle control process (S5) shown in Figure 2.

[0037] First, the 3D object detection unit 301 acquires an image obtained from the stereo camera or monocular camera from the image data buffer 203 (S31). Next, the 3D object detection unit 301 detects a 3D object in the image (S32). Next, the 3D object size calculation unit 302 calculates the size of the detected 3D object (S33).

[0038] Next, the object identification unit 303 performs identification processing of the detected object (S41) and determines whether the object is a pedestrian, which is the object to be identified. The lateral speed calculation unit 304 calculates the pedestrian's lateral speed if the object is a pedestrian (S42). Next, the control processing unit 108 performs vehicle control processing (S5) and terminates this process.

[0039] The lateral velocity calculated in step S42 is the velocity in the x-direction when the vehicle's longitudinal axis is the z-axis and its lateral axis is the x-axis, and is the lateral component of the pedestrian's relative velocity to the vehicle. If a pedestrian is present in the vehicle's lane, or if it is determined that a pedestrian located outside the vehicle's lane is moving at the aforementioned lateral velocity and is about to jump into the vehicle's lane, the vehicle control process (S16) will issue a warning or apply brakes to assist in the safety of the driver and pedestrian.

[0040] <Explanation of the process for calculating pedestrian lateral speed> Next, we will explain an example of a process that calculates the lateral speed of a pedestrian using images obtained from a stereo camera.

[0041] Figure 4 shows the results of detecting three-dimensional objects based on camera images using the three-dimensional object detection process (S3) shown in Figure 2. Here, we will explain the details of the three-dimensional object detection process using the image 401 acquired from the stereo camera by the three-dimensional object detection unit 301 shown in Figure 2 as an example.

[0042] When the 3D object detection unit 301 detects a 3D object in the image 401, it sets up multiple windows 402 and 403 of different sizes in the image 401. The windows 402 and 403 are areas in which the 3D object identification unit 303 identifies whether or not the 3D object detected by the 3D object detection unit 301 at each position in the image 401 is a pedestrian.

[0043] As described above, the distance to a three-dimensional object is determined by the parallax processing (S2) shown in Figure 2. Therefore, the three-dimensional object detection unit 301 can accurately identify three-dimensional objects at different distances by using a large window 402 for three-dimensional objects close to the vehicle and a small window 403 for three-dimensional objects far from the vehicle.

[0044] The 3D object detection unit 301 slides windows 402 and 403 within the image 401 and performs object identification processing (scanning processing) at each position after sliding by a predetermined amount. When a 3D object is detected, rectangular detection regions 404 and 405 are shown in the image 401 at the location where the 3D object was detected.

[0045] The detection regions 404 and 405, which are the detection results of the 3D object detection process (S3), indicate the areas on the image where 3D objects such as pedestrians and vehicles existing in 3D space are projected. The 3D object identification unit 303 reduces processing costs by identifying whether or not a 3D object is a pedestrian based on the detection regions 404 and 405.

[0046] The detection areas 404 and 405 may be rectangular or irregular in shape, changing according to parallax and distance. However, they are generally treated as rectangles to facilitate computer processing in the 3D object recognition process (S4). Therefore, the details of each process below will be explained assuming that the 3D object recognition unit 303 uses a rectangular detection area 404 to identify a 3D object.

[0047] Next, we will explain the conventional process for calculating lateral velocity and the calculation results. Figures 5A to 5C show examples of conventional speed calculation processes and calculation results. Pedestrian identification is performed based on the detection area 404 calculated from the image 401 obtained from the stereo camera.

[0048] Figure 5A shows an example of the right edge 501 of a pedestrian 500 as seen in the detection area 404. If the identification result is a pedestrian 500, the process proceeds to the lateral velocity calculation process (S15) shown in Figure 3. The conventional calculation processing unit refers to the left and right edges of the pedestrian 500 in order to calculate the lateral velocity of the pedestrian 500. The following explanation shows an example in which the right edge 501 of the pedestrian 500 is referred to. In the figure, the right edge 501 is represented by a thick solid curve as the right-side contour of the pedestrian 500.

[0049] Figure 5B shows an example of the horizontal position average 502 of the right edge 501 calculated from the image 401 of the t-frame. Conventional processing units, for example, obtain the horizontal position of each pixel of the right edge 501 and calculate the average value of these horizontal positions. The average horizontal position is represented as the horizontal position average 502 by a thick solid line. As the horizontal position average, for example, the average value of the horizontal position in the detection area 404 is calculated based on the horizontal position coordinates of multiple pixels sampled from the right edge 501.

[0050] Figure 5C shows an example of the lateral position average 502 of the right edge 501 calculated from image 401 in the (t+1) frame. In the (t+1) frame, the lateral position average 502 of the right edge 501 is obtained using the same process as in the t frame shown in Figure 5B. Here, if the pedestrian 500 was moving to the right, the amount of movement 504 from the lateral position average 502 to the lateral position average 503 between the t frame and the (t+1) frame is calculated.

[0051] Thus, the conventional processing unit calculates the average horizontal position 502 of the previous frame and the average horizontal position 502 of the current frame. Subsequently, the conventional processing unit calculates the amount of movement 504 of the average horizontal position 502 for each frame, and calculates the lateral velocity of the pedestrian 500 based on the time between frames and the amount of movement 504. The larger the amount of movement 504, the faster the pedestrian 500 is moving in the direction of the average horizontal position 502.

[0052] Next, we will explain the conventional problems that arise when calculating the horizontal average. Figures 6A to 6C show examples of images taken when a pedestrian 600 is standing near a tall, three-dimensional object such as a utility pole.

[0053] Figure 6A shows an example of the detection result when an object 601 that is not the target object is present near a pedestrian 600. The image captured in Figure 6A is considered the first frame. In the 3D object detection process (S3) shown in Figure 3, if an object 601 other than the pedestrian, such as a utility pole or signpost, is present near the pedestrian 600, an error occurs in which a detection area 602 larger than the pedestrian 600 is mistakenly detected. In this case, the detection area 602 includes the edge position 603 of the object 601, so noise is added to the edge position of the pedestrian 600. As a result, the horizontal position average 604, shown by the solid line, is calculated at a position different from the ideal horizontal position average 605, shown by the dashed line.

[0054] Figure 6B shows an example of the detection result for the second frame. The image captured in Figure 6B is considered the second frame. In both the first and second frames, the pedestrian 600 is actually stationary. However, let's assume that in the second frame, the non-target object 601 is no longer within the detection area, for example, due to the vehicle moving forward. In this case, only the edge 606 for the pedestrian 600 is calculated, and the edge 606 does not include the edge of the non-target object 601. Therefore, the conventional processing unit calculates the horizontal position average 607 for the edge 606.

[0055] Figure 6C shows an example of the average horizontal movement. Compared to the average horizontal movement of 604 calculated in the first frame, the movement of 608 calculated in the second frame (average horizontal movement of 607) is calculated to be larger. Due to this movement of 608, the pedestrian's lateral speed is incorrectly calculated to be larger than it actually is, even though the pedestrian 600 is stationary. One example of such a situation is when the vehicle is turning right or left, and at one point the pedestrian and the utility pole appear to overlap, but at the next point they no longer overlap. Another example is when the pedestrian is far away from the vehicle and appears to overlap with the utility pole, but as the vehicle approaches the pedestrian, they no longer overlap.

[0056] Thus, conventional methods for calculating movement distance have a problem where a pedestrian 600 that is actually stationary and located outside the lane is incorrectly judged as having stepped into the lane, leading to false alarms and incorrect braking. Therefore, it is necessary to correctly calculate the movement distance of a pedestrian even when an object 601 that is not the target object is present within the detection area.

[0057] Next, the process for calculating the amount of movement according to the first embodiment will be described. Figures 7A and 7B show examples of the calculation process for the amount of movement according to the first embodiment. In Figures 7A and 7B, we will explain the case where the pedestrian 600 is standing near a tall object such as a utility pole.

[0058] In the first embodiment, the 3D object detection unit 301 of the in-vehicle stereo camera device 100 first calculates the physical size of the 3D object from the size of the detection area and the distance on the image. Here, because the size of the 3D object (especially pedestrians) detected by the 3D object detection unit 301 is unstable, even if a detection area 602 larger than the pedestrian 600 is detected, as shown in Figure 6A, the lateral speed calculation unit 304 needs to suppress the calculation of an incorrect speed for the pedestrian 600. Therefore, two methods for the lateral speed calculation unit 304 to set the sampling range of edges used for lateral speed using the size of the 3D object will be described.

[0059] (First method for setting the edge sampling range) First, we will explain the edge sampling method (the first method) using a predetermined pedestrian size, with reference to Figures 7A and 7B.

[0060] Figure 7A shows an example of a detection area 602 that includes a pedestrian 600. Figure 7A shows an example where the first frame image has been acquired.

[0061] Since the pedestrian 600 was near an object 601 (such as a utility pole), the 3D object detection unit 301 assumes that it detected a detection area 602 larger than the pedestrian 600. Therefore, the lateral velocity calculation unit 304 refers to the physical quantity of the vertical width of the detection area 602 and samples edges within the area 701, excluding the portion above a predetermined value from the road surface, to calculate the lateral position average 702. The predetermined value of the height from the road surface is, for example, 2.0m. Alternatively, the predetermined value of the height from the road surface may be, for example, 2.1m, 2.2m, etc. The predetermined value of the height from the road surface is set based on the height of a person. The movement information calculation unit (lateral velocity calculation unit 304) sets the sampling range excluding the area in the image where the height from the road surface exceeds a predetermined value.

[0062] Figure 7B is a magnified view of the right side of pedestrian 600 as captured in the second frame. Figure 7B shows an example of how the second frame image was acquired.

[0063] In the detection area 602 of the second frame image, there are no non-target objects 601. Therefore, as explained with reference to Figure 6B, the lateral velocity calculation unit 304 calculates the lateral position average 607 using only the right edge of the pedestrian 600. As a result, the amount of movement 703 from the lateral position average 702 calculated in the first frame to the lateral position average 607 calculated in the second frame is much smaller than the amount of movement 608 calculated in Figure 6C. In this way, the three-dimensional object detected from images of multiple frames is identified as a pedestrian 600, so the lateral velocity calculation unit 304 can accurately calculate the lateral velocity of the pedestrian 600.

[0064] (A second method for setting the sampling range of edges) Next, we will explain the edge sampling method using the identification results (the second method) with reference to Figures 8A and 8B.

[0065] Figure 8A shows an example of a detection area 710 that includes a pedestrian 600. Figure 8A shows an example where the first frame image has been acquired.

[0066] The 3D object identification unit 303 stores the identification result of the 3D object identified in the first frame in the storage unit 106, linked to the 3D object size calculated by the 3D object size calculation unit 302, so that the 3D object identification unit 303 can refer to it in the processing of the next frame. Next, the 3D object identification unit 303 calculates the identification score of the detection area 710 in the 3D object identification process (S41) shown in Figure 3. For example, if the minimum value of the identification score is 0 points and the maximum value is 100 points, as shown in Figure 8A, if the detection area 710 contains a large proportion of the pedestrian 600 that is the target of identification, a high identification score will be assigned to this detection area 710.

[0067] If the object detection process by the 3D object detection unit 301 is stable, the accuracy of object identification by the 3D object identification unit 303 will be high. For example, if the identification score calculated on a frame-by-frame basis for the detection area 710 is high, the object detection process is stable. Therefore, it can be assumed that the object size calculated by the 3D object size calculation unit 302 based on the detection area 710 is also stable. In the first frame image shown in Figure 8A, we assume that the identification score for the pedestrian 600 was high due to the stable detection area 710.

[0068] Figure 8B shows an example where a detection area 602 larger than the object to be identified was acquired. Figure 8B shows an example where the second frame image was acquired.

[0069] For example, suppose that after the 3D object detection unit 301 acquires the image for the first frame, the detection area setting becomes unstable, and in the next frame, the 3D object detection unit 301 acquires a detection area 602 that is larger than the object to be identified (pedestrian 600). In this case, the edge position 603 of the non-target object 601 is included above the detection area 602, and the blank area other than the pedestrian 600 is larger than the detection area 710 shown in Figure 8A. Therefore, the lateral velocity calculation unit 304 cannot calculate the desirable lateral position average for the edge of the pedestrian 600.

[0070] Therefore, the movement information calculation unit (lateral velocity calculation unit 304) sets the sampling range based on the identification result indicating that the object is the object to be identified, and the size of the object calculated from the image of the frame in which the identification result was obtained. For example, the lateral velocity calculation unit 304 sets the size of the object 704 in the detection area 710 of the first frame, which it determined to have a high identification score in the image of Figure 8A, as the sampling range of the edge used for calculating the lateral velocity in the next frame, the second frame. By calculating the lateral position average 705 using the sampling range set based on the size of the object 704, the lateral velocity calculation unit 304 can suppress the calculation of the pedestrian's movement amount being larger than the actual movement amount.

[0071] In the second method, since a discrimination score is used, for example, even if the detection process becomes unstable and a large detection region 602 is obtained in the second frame image, the edges of the three-dimensional object can be sampled using the sampling range set based on the detection region 710 with a high discrimination score in the first frame image.

[0072] Furthermore, in the second method, the predetermined amount from the road surface, as explained in the first method, is not set. Therefore, the height of the detection area 710 with a high identification score is variable, and may be 1.5m or 1.8m. However, since it is not necessary to pre-set the predetermined amount from the road surface, it is more versatile than the first method.

[0073] By using the first or second method described above, the lateral speed calculation unit 304 can suppress an increase in the average lateral movement amount 703 and prevent erroneous speed calculation for stationary pedestrians 600.

[0074] Figure 9 is a flowchart showing a detailed example of the 3D object recognition process (S4) in Figure 2.

[0075] First, the 3D object identification unit 303 performs pedestrian identification processing (S41) on the detection area obtained in the 3D object detection processing (S3) as shown in Figure 3 (S51). In the pedestrian identification processing, the 3D object identification unit 303 determines whether the 3D object detected in the detection area is a pedestrian or not based on the identification results of multiple frames (S52). If the 3D object identification unit 303 determines that the object to be identified is not a pedestrian (NO in S52), it terminates this process.

[0076] On the other hand, if the object identification unit 303 determines that the object to be identified is a pedestrian (YES in S52), it proceeds to the lateral velocity calculation process (S42). In the lateral velocity calculation process (S42), the lateral velocity calculation unit 304 calculates the left and right edges of the pedestrian (S61).

[0077] Next, the lateral velocity calculation unit 304 calculates the three-dimensional object size of the pedestrian (S62). In the three-dimensional object size calculation process, the lateral velocity calculation unit 304 calculates the physical quantity of the three-dimensional object size (e.g., height) based on the detection area and distance on the image. Note that the three-dimensional object size calculated in step S62 is the size of the pedestrian, but the three-dimensional object size calculated by the three-dimensional object size calculation unit 302 shown in Figure 2 is the size of a three-dimensional object that may or may not be a pedestrian. Alternatively, instead of calculating the three-dimensional object size in step S62, the lateral velocity calculation unit 304 may obtain the three-dimensional object size calculated in step S33 in Figure 3 from the three-dimensional object size calculation unit 302.

[0078] Next, the lateral velocity calculation unit 304 samples the left and right edges of the object using the object size calculated in step S62 (S63). Then, the lateral velocity calculation unit 304 calculates the average of the lateral positions of the sampled left and right edges (S64).

[0079] Next, the lateral velocity calculation unit 304 calculates the average lateral movement of the left and right edges in the current frame based on the average lateral position of the left and right edges calculated in past frames and the average lateral position of the left and right edges in the current frame (S65). Finally, the lateral velocity calculation unit 304 calculates the pedestrian's lateral velocity (S66) and terminates this process.

[0080] <Example of computer hardware configuration> Next, the hardware configuration of the computer 800 that constitutes each component of the in-vehicle stereo camera system 100 will be described.

[0081] Figure 10 is a block diagram showing an example of the hardware configuration of computer 800. Computer 800 is an example of hardware used as a computer capable of operating as the in-vehicle stereo camera device 100 according to this embodiment. In the in-vehicle stereo camera device 100 according to this embodiment, computer 800 (computer) executes a program to realize various processes performed in cooperation with the functional blocks shown in Figures 1 and 2.

[0082] Computer 800 comprises a CPU (Central Processing Unit) 801, a ROM (Read Only Memory) 802, and a RAM (Random Access Memory) 803, each connected to a bus 804. Furthermore, computer 800 includes non-volatile storage 805 and a network interface 806.

[0083] The CPU 801 reads the program code of the software that implements each function according to this embodiment from the ROM 802, loads it into the RAM 803, and executes it. Variables and parameters that occur during the calculation process of the CPU 801 are temporarily written to the RAM 803, and these variables and parameters are read out by the CPU 801 as appropriate. However, an MPU (Micro Processing Unit) or GPU (Graphics Processing Unit) may be used instead of the CPU 801, or the CPU 801 and GPU (Graphics Processing Unit) may be used in combination.

[0084] Examples of non-volatile storage 805 include HDDs (Hard Disk Drives), SSDs (Solid State Drives), flexible disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, magnetic tapes, or non-volatile memory. This non-volatile storage 805 stores the OS (Operating System), various parameters, and programs necessary for the computer 800 to function. ROM 802 and non-volatile storage 805 store programs and data necessary for the CPU 801 to operate, and are used as an example of a non-transient storage medium readable by a computer that stores programs executed by the computer 800.

[0085] The network interface 806 can use, for example, a NIC (Network Interface Card), and various types of data can be sent and received between devices via a LAN (Local Area Network), dedicated line, etc., connected to the terminals of the NIC.

[0086] The in-vehicle stereo camera device 100 according to the first embodiment described above, when using a stereo camera as a sensor to detect external information, calculates disparity information from the image captured by the stereo camera and acquires an image of a three-dimensional object. Subsequently, when the in-vehicle stereo camera device 100 identifies the three-dimensional object identified from the image as a pedestrian, it calculates the left and right edges of the pedestrian and calculates the size of the three-dimensional object based on the left and right edges. Here, when the three-dimensional object is identified as a pedestrian, the sampling range is set based on the size of the three-dimensional object, so that the edges of non-target objects that are close to the pedestrian in the image are not sampled. As a result, even when the detection of the three-dimensional object to be identified is unstable, a highly accurate lateral velocity can be calculated. Furthermore, by improving the accuracy of calculating the lateral velocity, the accuracy of warnings or brake control to avoid collisions with three-dimensional objects by the vehicle is improved, leading to improved safety assistance for drivers and pedestrians.

[0087] Furthermore, the system identifies whether or not a three-dimensional object is the object to be identified based on images captured in multiple frames. Therefore, even if the three-dimensional object was captured near an object that is not the object in a previous frame, the three-dimensional object identification unit 303 can accurately identify that the three-dimensional object is the object to be identified if the three-dimensional object is captured further away from the object in a subsequent frame.

[0088] In the first method, the height from the road surface in the image is set to the average height of the pedestrian to be identified, thereby setting the sampling range to exclude areas exceeding the height from the road surface. As a result, the lateral velocity calculation unit 304 can prevent the lateral position values ​​calculated based on non-target objects that are taller than the identified object from being included in the lateral position average value due to the sampling of non-target objects that are taller than the identified object.

[0089] In the second method, the identification result, which indicates that a three-dimensional object is the target of identification, is expressed as an identification score. A higher identification score indicates that the detection process for the target of identification is more stable. Therefore, the size of the three-dimensional object calculated based on the detection area with a high identification score also becomes stable, and the sampling range is set based on that three-dimensional object size.

[0090] Furthermore, the in-vehicle stereo camera device 100 according to this embodiment can accurately calculate the movement of pedestrians even when they are walking in a location other than a crosswalk.

[0091] Furthermore, while the embodiment described above identifies pedestrians, at least one of the following may also be identified: a person riding a bicycle, a person riding a motorcycle, or a person riding a personal mobility vehicle.

[0092] Furthermore, in the embodiment described above, an example was shown in which the lateral velocity calculation unit 304 equally sampled either the left or right edge of the pedestrian 600, but the sampled information may be weighted. For example, when the pedestrian 600 is moving, the lower body moves more than the upper body, so errors are likely to occur in the lateral position average due to the movement of the lower body. Therefore, the lateral velocity calculation unit 304 can accurately determine the lateral position average of the pedestrian 600 by weighting the sampled information of the upper body of the pedestrian 600 more than that of the lower body.

[0093] [Second Embodiment] Next, an example of the configuration of an in-vehicle camera device according to the second embodiment of the present invention will be described with reference to Figure 11. The in-vehicle camera device according to the second embodiment uses a monocular camera and millimeter-wave radar as sensors, instead of the stereo camera used as a sensor in the first embodiment.

[0094] Figure 11 is a block diagram showing an example configuration of the in-vehicle camera device 900. The in-vehicle camera device 900 includes a monocular camera 901, a millimeter-wave radar 902, an image data buffer 903, a distance measurement data buffer 904, and a CAN IF 910. The in-vehicle camera device 900 has the same functional blocks as the in-vehicle stereo camera device 100 shown in Figure 1. Therefore, the monocular camera 901 is connected to the image input IF 103 (see Figure 1), and the image data input through the image input IF 103 is stored in the storage unit 106 via the internal bus 109. In addition, an input IF (not shown) for the millimeter-wave radar 902 is provided, and the data output by the millimeter-wave radar 902 is input through this input IF and stored in the storage unit 106 via the internal bus 109.

[0095] The monocular camera (monocular camera 901) is an example of an imaging unit having one image sensor, and it captures the environment in front of the vehicle. The image data of the image captured by the monocular camera 901 is processed (S21) by the image processing unit 104. The content of the image processing (S21) is the same as the image processing (S1) shown in Figure 2. The image data after image processing is buffered in the image data buffer 903 provided in the storage unit 106. Unlike the first embodiment described above, the imaging unit is not configured as a stereo camera, so the parallax processing (S2) shown in Figure 2 is not performed in the in-vehicle camera device 900 according to the second embodiment.

[0096] The range measuring unit (millimeter-wave radar 902) measures the distance to three-dimensional objects. For example, the millimeter-wave radar 902 emits millimeter-wave radio waves in front of the vehicle and measures the distance to an object that reflects the radio waves, the speed of the object, etc. The data output by the millimeter-wave radar 902 is processed by the arithmetic processing unit 105 (S22) for range measurement. The processed range measurement data is buffered in the range measurement data buffer 904 provided in the storage unit 106.

[0097] Subsequently, a three-dimensional object detection process (S23) is performed using the image data read from the image data buffer 903 and the distance measurement data read from the distance measurement data buffer 904. The three-dimensional object detection process (S23) corresponds to the three-dimensional object detection process (S3) shown in Figure 2, and the detection areas 404 and 405 shown in Figure 4 are switched and used according to the distance measurement data, and three-dimensional objects are detected.

[0098] The three-dimensional object identification process (S24) and vehicle control process (S25) following the three-dimensional object detection process (S23) correspond to the three-dimensional object identification process (S4) and vehicle control process (S5) shown in Figures 2 and 3. For example, the three-dimensional object detection unit (three-dimensional object detection unit 301) detects three-dimensional objects at a specific distance based on the image input from the image sensor. The three-dimensional object identification unit (three-dimensional object identification unit 303) identifies the detected three-dimensional objects. In the three-dimensional object size calculation process (S33) of the three-dimensional object detection process (S3) shown in Figure 3, the three-dimensional object size calculation unit 302 can calculate the size of the three-dimensional object based on the image obtained from the monocular camera 901 and the distance and detection size obtained by the millimeter-wave radar 902.

[0099] The CAN IF910 outputs information from the in-vehicle camera device 900 to the vehicle's control system (not shown) via the in-vehicle network CAN110 shown in Figure 1.

[0100] Furthermore, the present invention can also be applied to devices equipped with sensors capable of simultaneously measuring images and distance, in addition to the monocular camera 901 and the millimeter-wave radar 902.

[0101] In the in-vehicle camera device 900 according to the second embodiment described above, similar to the in-vehicle stereo camera device 100 according to the first embodiment, when a three-dimensional object detected in the image is identified as a pedestrian, the lateral speed can be calculated using the first and second methods described above. Therefore, even if there is an object that is not the target, such as a tall utility pole, near the pedestrian, it is possible to accurately calculate the pedestrian's lateral speed.

[0102] It should be noted that the present invention is not limited to the embodiments described above, and various other applications and modifications can be taken as long as they do not depart from the gist of the present invention as described in the claims. For example, the embodiments described above are detailed and specific explanations of the configuration of the environmental recognition device in order to clearly explain the present invention, and are not necessarily limited to having all the configurations described. Furthermore, it is possible to replace some of the configurations of the embodiments described here with the configurations of other embodiments, and it is also possible to add the configurations of other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace some of the configurations of each embodiment with other configurations. Furthermore, the control lines and information lines shown are those deemed necessary for explanatory purposes, and not all control lines and information lines are necessarily shown in the actual product. In reality, it is safe to assume that almost all components are interconnected. [Explanation of Symbols]

[0103] 100...In-vehicle stereo camera device, 101...Left camera, 102...Right camera, 103...Image input interface, 104...Image processing unit, 105...Calculation processing unit, 106...Storage unit, 108...Control processing unit, 109...Internal bus, 110...In-vehicle network CAN, 203...Image data buffer, 301...3D object detection unit, 302...3D object size calculation unit, 303...3D object identification unit, 304...Lateral speed calculation unit

Claims

1. A three-dimensional object detection unit sets a detection area for detecting an object to be identified in an image captured by an imaging unit, and detects three-dimensional objects in the image based on the detection area. A three-dimensional object size calculation unit calculates the size of the three-dimensional object detected in the detection area using distance information from the device to the three-dimensional object calculated from the aforementioned image, A three-dimensional object identification unit identifies the three-dimensional object detected in the detection area as an object to be identified based on the images of a plurality of consecutive frames, The system includes a movement information calculation unit that limits the sampling range of the three-dimensional object based on its size, calculates the average value of the horizontal coordinates of the points sampled from the edges of the three-dimensional object for each frame as edge information of the three-dimensional object to be identified, and calculates the speed at which the three-dimensional object moves laterally as movement information based on the edge information. Environment recognition device.

2. The movement information calculation unit sets the sampling range excluding areas in the image where the height from the road surface exceeds a predetermined value. The environmental recognition device according to claim 1.

3. The movement information calculation unit sets the sampling range based on the identification result indicating that the three-dimensional object is the object to be identified, and the size of the three-dimensional object calculated from the image of the frame in which the identification result was obtained. The environmental recognition device according to claim 1.

4. The movement information calculation unit samples either the left or right edge of the three-dimensional object and calculates the average value of the horizontal coordinates of the sampled edge. The environmental recognition device according to claim 2.

5. The imaging unit is a stereo camera having two image sensors, The system includes an image processing unit that calculates the parallax of the three-dimensional object based on the images input from each of the image sensors, The aforementioned three-dimensional object detection unit detects a three-dimensional object that is the same distance from the vehicle on which the device and the stereo camera are mounted, The three-dimensional object identification unit identifies three-dimensional objects that are the same distance apart. The environmental recognition device according to claim 4.

6. The imaging unit is a monocular camera having one image sensor, It is equipped with a distance measuring unit for measuring the distance to the aforementioned three-dimensional object, The three-dimensional object detection unit detects the three-dimensional object at a specific distance based on the image input from the image sensor. The three-dimensional object identification unit identifies the detected three-dimensional object. The environmental recognition device according to claim 4.

7. The three-dimensional object to be identified is at least one of the following: a pedestrian, a person riding a bicycle, a person riding a motorcycle, or a person riding a personal mobility vehicle. The environmental recognition device according to claim 4.

8. The aforementioned lateral direction is the direction in which a pedestrian moves across the front of the vehicle on which the device is mounted, or across the lane in which the vehicle is traveling. The environmental recognition device according to claim 4.

9. The imaging unit sets a detection area for detecting an object to be identified in the image it has captured, and detects a three-dimensional object in the image based on the detection area. A step of calculating the size of the three-dimensional object detected in the detection area using the distance information from the device to the three-dimensional object calculated from the aforementioned image, A step of identifying that the three-dimensional object detected in the detection area is the object to be identified, based on the images of a plurality of consecutive frames, The steps include: limiting the sampling range of the three-dimensional object based on its size; calculating the average value of the horizontal coordinates of the points sampled from the edges of the three-dimensional object for each frame as information about the edges of the three-dimensional object to be identified; and calculating the speed at which the three-dimensional object moves laterally as movement information based on the edge information. Environmental recognition method.

Images