Environment recognition device and environment recognition method

Abstract

An environment recognition device is provided with: a three-dimensional object detection unit that sets a detection region for detecting an object to be recognized for an image captured by an imaging unit, and detects a three-dimensional object appearing in the image on the basis of the detection region; a three-dimensional object size calculation unit that calculates the size of the three-dimensional object detected in the detection region using information on the distance from the device to the three-dimensional object calculated from the image; a three-dimensional object recognition unit that recognizes that a three-dimensional object detected in the detection region is an object to be recognized; and a movement information calculation unit that defines the sampling range of the three-dimensional object on the basis of the size of the three-dimensional object, and calculates movement information for the lateral movement of the three-dimensional object as the recognition target on the basis of the edge information of the three-dimensional object as the recognition target.

Description

Technical Field

[0001] The present invention relates to an environment recognition device and an environment recognition method. Background Art

[0002] In recent years, with the widespread use of in-vehicle sensor systems, there has been a demand for improved performance in collision safety features for pedestrians. The New Car Assessment Programme (NCAP) has been established to evaluate vehicle safety (automotive assessment). For this purpose, various regions use programs such as Euro NCAP and Japan NCAP. Various evaluation items are being actively added or changed in automotive assessments. As the NCAP continues to evolve, it is necessary to address the need for improved performance in collision safety features. An example of evaluating the performance of collision safety features is the evaluation of automatic braking for unexpected pedestrians.

[0003] To determine if a pedestrian has suddenly appeared, the vehicle's ECU first identifies the pedestrian based on images captured by the vehicle's onboard camera and calculates their lateral speed. Next, if the ECU determines, based on the pedestrian's speed and direction, that the pedestrian is moving toward the vehicle's lane, it applies collision damage mitigation braking (AEB).

[0004] Patent document 1 states: "Generate a continuous contour line with the peak point of the edge toward the top of the image, classify it into contour line groups based on the horizontal movement of the contour line between the camera images, and based on the horizontal movement of the contour line group and the height direction dimension of the contour line group, determine with high precision whether there is a pedestrian who is highly dangerous and suddenly rushes into the driving path of the vehicle." Prior art literature Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2012-203884. Summary of the Invention Problems to be solved by the invention

[0006] Traditionally, 3D object recognition processing implemented in the ECU (Electronic Control Unit) presupposes accurate detection of a 3D object. This reduces processing costs by maintaining recognition performance within the detection area for that 3D object. However, in reality, accurate 3D object detection may not be possible due to factors such as nighttime, the color of clothing and the background, and distance. For example, if a pedestrian is near a utility pole or sign, the detected area may be larger than the actual size of the pedestrian.

[0007] In calculating the lateral velocity of pedestrians, there is a method that uses the edges of pedestrians within a detection area. In the technology described in Patent Document 1, edges (contour lines) are extracted from an image, and based on the size of the edges, it is determined whether the three-dimensional object detected in the detection area is a pedestrian. If the judgment result is a pedestrian, the pedestrian's lateral velocity is calculated. If the judgment result is not a pedestrian, the pedestrian is treated as a non-pedestrian. However, in the technology described in Patent Document 1, due to the background conditions of the pedestrian, it is not possible to extract accurate edges. Therefore, the edges may be treated as non-pedestrians, and there is a risk that the accurate lateral velocity cannot be calculated. In addition, in the method that uses the edges within the detection area, as described above, when an area larger than the actual pedestrian size is detected, an incorrect lateral velocity may be calculated for a stationary pedestrian. If an incorrect lateral velocity is calculated in this way, there is a problem in the ECU's vehicle control processing that an alarm may be issued or emergency braking control may be performed even if there is no danger of sudden run-out.

[0008] The present invention has been made in view of such circumstances, and an object thereof is to calculate movement information of a three-dimensional object to be recognized with high accuracy. Technical means to solve the problem

[0009] The environment recognition device of the present invention comprises: a three-dimensional object detection unit, which sets a detection area for detecting a recognition object in an image captured by a camera unit and detects a three-dimensional object appearing in the image based on the detection area; a three-dimensional object size calculation unit, which uses information on the distance from the device to the three-dimensional object calculated based on the image to calculate the size of the three-dimensional object detected in the detection area; a three-dimensional object recognition unit, which recognizes that the three-dimensional object detected in the detection area is a recognition object; and a movement information calculation unit, which limits a sampling range of the three-dimensional object based on the size of the three-dimensional object and calculates movement information of the lateral movement of the three-dimensional object as the recognition object based on edge information of the three-dimensional object as the recognition object. Effects of the Invention

[0010] According to the present invention, by limiting the sampling range of a three-dimensional object based on the size of the three-dimensional object, even when the detection of the three-dimensional object is unstable, the movement information of the three-dimensional object to be recognized can be calculated with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a block diagram showing an example of the overall configuration of a vehicle-mounted stereo camera device according to the first embodiment of the present invention. Figure 2 This is a flowchart and a configuration diagram showing an example of processing performed by each functional unit of the vehicle-mounted stereo camera device according to the first embodiment of the present invention. Figure 3 This is a flowchart showing an example of processing for calculating the lateral velocity of a three-dimensional object, which is the basis of the present invention. Figure 4 This figure shows the results of three-dimensional objects detected in the three-dimensional object detection process based on the camera image according to the first embodiment of the present invention. Figure 5A : is a diagram showing an example of the right edge of a pedestrian appearing in the detection area. Figure 5B : is a diagram showing an example of the average value of the horizontal position of the right edge calculated from the t-th frame image. Figure 5C 1 is a diagram showing an example of the average value of the right edge horizontal position calculated based on the (t+1)th frame image. Figure 6A 1 is a diagram showing an example of detection results when a non-target object exists near a pedestrian. Figure 6B This is a diagram showing an example of the detection result of the second frame. Figure 6C 1 is a diagram showing an example of the amount of shift of the average value of the horizontal position. Figure 7A is an example of a detection area that includes a pedestrian. Figure 7B This is an enlarged view of the right side of the pedestrian captured in the second frame. Figure 8A FIG. 1 is a diagram showing an example of a detection area including a pedestrian. Figure 8B This figure shows an example in which a detection area larger than the object is acquired. Figure 9 This is a flowchart showing an example of detailed processing of the three-dimensional object recognition process according to the first embodiment of the present invention. Figure 10 This is a block diagram showing an example of the hardware configuration of a computer according to the first embodiment of the present invention. Figure 11 This is a block diagram showing a configuration example of a vehicle-mounted camera device according to a second embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following describes embodiments of the present invention with reference to the accompanying drawings. In this specification and the accompanying drawings, components having substantially the same function or configuration are designated by the same reference numerals to avoid redundant description. For example, the present invention is applicable to a vehicle control computing device capable of communicating with an onboard ECU (Electronic Control Unit) for an Advanced Driver Assistance System (ADAS) or Autonomous Driving (AD).

[0013] [First embodiment] Figure 1 This is a block diagram showing an example of the overall configuration of the vehicle-mounted stereo camera device 100 according to the first embodiment of the present invention.

[0014] The on-board stereo camera device 100, mounted on a vehicle (not shown), is an example of an environment recognition device that recognizes the external environment based on image information captured in the vehicle's direction of travel, i.e., in front of the vehicle. Based on this image information, the on-board stereo camera device 100 recognizes, for example, road lines, pedestrians, vehicles, other three-dimensional objects, traffic lights, signs, and illuminated lamps. Furthermore, based on the results of the recognition of the external environment, the on-board stereo camera device 100 adjusts the vehicle (hereinafter referred to as the "host vehicle") by braking and steering. The on-board stereo camera device 100 safely controls the host vehicle by calculating the movement information (movement speed, movement distance, or movement direction) of laterally moving pedestrians. In the following description, the direction in which a pedestrian crosses in front of the vehicle carrying the on-board stereo camera device 100 or in the vehicle's lane is referred to as "lateral direction." Lateral direction can be perpendicular to the vehicle's direction of travel or the direction of its lane, or it can be oblique.

[0015] The vehicle-mounted stereo camera device 100 includes a left camera 101 , a right camera 102 , an image input I / F (InterFace) 103 , an image processing unit 104 , an arithmetic processing unit 105 , a storage unit 106 , a CAN (Controller Area Network) IF 107 , a control processing unit 108 , and an internal bus 109 .

[0016] The left camera 101 and the right camera 102 are respectively arranged on the left and right sides of the vehicle, and the camera elements carried by each camera capture the environment in front of the vehicle to obtain image information. In the following description, the left camera 101 and the right camera 102 are also referred to as "stereo cameras" and "camera units". In addition, the image information output by the camera elements of the stereo camera is referred to as "image". The camera unit (left camera 101, right camera 102) is a stereo camera with two camera elements. Figure 1 In FIG, the stereo camera is composed of two left cameras 101 and right cameras 102, each of which has one imaging element. However, even if a single camera has two imaging elements arranged horizontally, the camera can be treated as a stereo camera.

[0017] The image input interface 103 is connected to the imaging element of the stereo camera. The image input interface 103 then controls the stereo camera's imaging operation to capture images captured by the stereo camera. Image data captured by the image input interface 103 is transmitted to various components via the internal bus 109. The image processing unit 104 and the arithmetic processing unit 105 then perform predetermined processing on the image data, and the image data, including the intermediate or final results of the processing, is stored in the storage unit 106.

[0018] The image processing unit 104 compares the first image obtained from the imaging element of the left camera 101 and the second image obtained from the imaging element of the right camera 102, and performs image correction such as correction of device-specific deviation caused by the imaging element and noise interpolation on each image. In addition, the image processing unit 104 stores the corrected image in the storage unit 106. Furthermore, the image processing unit (image processing unit 104) calculates the parallax of the three-dimensional object based on the first image and the second image input from the respective imaging elements on the left and right. At this time, the image processing unit 104 calculates the corresponding position between the first image and the second image to calculate the parallax information. Thereafter, the image processing unit 104 stores the parallax information in the storage unit 106 in the same manner as the process of storing the corrected image in the storage unit 106. In the following description, when the first image and the second image are not distinguished, they are simply referred to as "images".

[0019] The processing unit 105 uses the images (captured by the left camera 101 or the right camera 102) and parallax information (distance information between points on the image) stored in the storage unit 106 to perform recognition processing for various three-dimensional objects necessary for identifying the vehicle's surrounding environment. Examples of these three-dimensional objects include people, vehicles, other obstacles, traffic lights, signs, and vehicle taillights and headlights. The recognition results and a portion of the intermediate calculation results obtained by the processing unit 105 are stored in the storage unit 106. The control unit 108 uses the recognition results of the images stored in the storage unit 106 to perform vehicle control calculations.

[0020] The control processing unit 108 uses the image and the recognition results of various three-dimensional objects from the calculation processing unit 105 to calculate the vehicle control policy. The vehicle control policy calculated by the control processing unit 108, as well as a portion of the three-dimensional object recognition results obtained by the calculation processing unit 105, are transmitted to the in-vehicle network CAN 110 via the CAN interface 107. The vehicle is then braked using the accelerator and brake (not shown). Furthermore, the control processing unit 108 monitors these operations for abnormal operation of each processing unit and errors in data transmission, thereby preventing abnormal operation.

[0021] The image processing unit 104 is connected to the image input interface 103, the arithmetic processing unit 105, the storage unit 106, and the control processing unit 108 via an internal bus 109. Furthermore, the image processing unit 104 is connected to the CAN interface 107 of the in-vehicle network CAN 110 located outside the vehicle-mounted stereo camera device 100 via the internal bus 109. As described below, the control processing unit 108, the image processing unit 104, the arithmetic processing unit 105, the storage unit 106, and the CAN interface 107 are composed of one or more computer units.

[0022] The storage unit 106 is constituted by, for example, a memory that stores images processed by the image processing unit 104 , scan results obtained by scanning images by the arithmetic processing unit 105 , and the like. The CAN IF 107 performs input and output with an external in-vehicle network. To this end, the CAN IF 107 outputs information output from the in-vehicle stereo camera device 100 to a control system (not shown) of the vehicle via the in-vehicle network CAN 110 .

[0023] Figure 2 It is a flowchart and a configuration diagram showing an example of processing performed in each functional unit of the vehicle-mounted stereo camera device 100. Figure 2 FIG. 4 shows an image recognition process as an example of an environment recognition method performed in the vehicle-mounted stereo camera device 100. Figure 2 In each of the processes shown, the corresponding Figure 1 Each processing is described using the reference numerals of the functional units shown.

[0024] First, the stereo cameras (left camera 101 and right camera 102) of the vehicle-mounted stereo camera device 100 capture images. The image processing unit 104 performs image processing (S1) on the image data 201 captured by the left camera 101 and the image data 202 captured by the right camera 102, for example, correction to absorb the inherent characteristics of the imaging elements of the stereo cameras. The image processing results are stored in the image data buffer 203. The image data buffer 203 is set Figure 1 The storage unit 106 has a function of temporarily storing image-related data.

[0025] Next, the image processing unit 104 compares the two corrected images read from the image data buffer 203. At this point, the image processing unit 104 compares the two images to perform parallax processing (S2) to obtain parallax information between the images (referred to as "left and right images") output by the stereo camera's imaging elements. Based on the parallax information between the left and right images, it is determined where a point of interest on a three-dimensional object corresponds to on the stereo camera image. This allows the distance to the object to be determined based on the principle of triangulation.

[0026] As mentioned above, image processing (S1) and parallax processing (S2) are Figure 1 The image processing unit 104 performs the processing, and the processed image, parallax information, and distance to the object are stored in the storage unit 106.

[0027] Next, the 3D object detection unit 301 of the processing unit 105 uses the parallax information obtained in the parallax processing (S2) to perform a 3D object detection process (S3) for detecting a 3D object in a three-dimensional space. In the 3D object detection process (S3), the 3D object detection unit (3D object detection unit 301) performs processing to detect a 3D object that has the same distance from the device and the vehicle equipped with the stereo camera. In the 3D object detection process (S3), in addition to the following, Figure 3 In addition to the image acquisition process ( S31 ) and the three-dimensional object detection process ( S32 ) performed by the three-dimensional object detection unit 301 , the three-dimensional object size calculation unit 302 also includes a process of calculating the size of the three-dimensional object.

[0028] In the 3D object detection process (S3), the 3D object detection unit (3D object detection unit 301) obtains the image captured by the camera unit (left camera 101, right camera 102), sets a detection area for detecting the recognition object for the image, and detects the 3D object ( Figure 3The three-dimensional object size calculation unit (three-dimensional object size calculation unit 302) uses the information of the distance from the device to the three-dimensional object calculated based on the image to calculate the size of the three-dimensional object detected in the detection area ( Figure 3 S33).

[0029] Next, the 3D object recognition unit 303 of the processing unit 105 performs 3D object recognition processing (S4) using the image, parallax information, and distance to the object stored in the storage unit 106. In the 3D object recognition processing (S4), the 3D objects to be recognized include people, cars, other 3D objects, signs, traffic lights, taillights, etc. The details of the 3D object recognition processing (S4) are determined by the characteristics of the object and the processing time allowed by the system. In the 3D object recognition processing (S4), in addition to the following, Figure 3 In addition to the three-dimensional object recognition process ( S41 ) performed by the three-dimensional object recognition unit 303 , the process also includes a lateral velocity calculation process ( S42 ) performed by the lateral velocity calculation unit 304 .

[0030] In the 3D object recognition process (S4), the 3D object recognition unit (3D object recognition unit 303) identifies the 3D object detected in the detection area as a recognition target (S41). The 3D object recognized as a recognition target by the 3D object recognition unit (3D object recognition unit 303) is located at the same distance in both images. The 3D object recognition unit (3D object recognition unit 303) identifies the 3D object as a recognition target based on a plurality of consecutive frame images. The 3D object recognition unit 303 can identify the 3D object as a pedestrian, the recognition target, by, for example, matching the person's walking posture or human body shape.

[0031] The movement information calculation unit (lateral velocity calculation unit 304) limits the sampling range of the three-dimensional object based on the size of the three-dimensional object (pedestrian), and calculates the movement information (lateral velocity) of the lateral movement of the three-dimensional object to be recognized based on the edge information of the three-dimensional object to be recognized (S42). Here, the movement information calculation unit (lateral velocity calculation unit 304) calculates the average value of the horizontal coordinates of the points where the edges of the three-dimensional object are sampled as edge information for each frame, and calculates the lateral velocity of the three-dimensional object to be recognized as the movement information.

[0032] In addition, the movement information calculation unit (lateral speed calculation unit 304) samples the edges on either side of the three-dimensional object (pedestrian) and calculates the average horizontal coordinates of the points where the edges were sampled. For example, the lateral speed calculation unit 304 samples the edges within the range from the side close to the road surface (such as the toes or heels) to the side away from the road surface (such as the head or hat) of the three-dimensional object. If the identification object is a motorcycle, the edges are sampled within the range of the motorcycle wheel close to the road surface and the helmet away from the road surface. In addition, in countries where traffic flows on the left, the right edge of the pedestrian is sampled, while in countries where traffic flows on the right, the left edge of the pedestrian is sampled. However, the lateral speed calculation unit 304 can also arbitrarily change the sampled edge to the right or left side based on the direction in which the three-dimensional object (pedestrian) to be identified is predicted to enter the lane of the vehicle.

[0033] Next, the control processing unit 108 comprehensively considers the three-dimensional object recognition results and the vehicle's state (speed, steering angle, etc.) to perform vehicle control processing (S5). During this vehicle control processing (S5), the control processing unit 108, for example, issues a warning to the vehicle occupants and performs braking operations such as braking the vehicle and adjusting the steering angle. Furthermore, the control processing unit 108 determines a policy for object avoidance control using braking and outputs the result to the in-vehicle network CAN 110 via the CAN IF 107.

[0034] Figure 2 The three-dimensional object recognition process (S4) shown in Figure 1 The vehicle control process (S5) is performed in the calculation processing unit 105. Figure 1 The control processing unit 108 performs the output to the vehicle network CAN 110 in the CAN IF 107. These processing and functional units are composed of, for example, one or more computer units and are configured to be able to exchange data with each other.

[0035] In addition, through the disparity processing (S2) performed by the image processing unit 104, the disparity or distance of each pixel in the left and right images can be obtained. Therefore, in the three-dimensional object detection processing (S3) performed by the three-dimensional object detection unit 301, pixels with the same disparity or distance are grouped as three-dimensional objects in three-dimensional space. In addition, in the three-dimensional object detection processing (S3), since the category of the three-dimensional object is not detected, if there is a person standing near a tree, the tree and the person may be grouped together. In addition, if multiple people are standing at the same distance, the multiple people may be grouped together. After the three-dimensional object detection processing (S3), the three-dimensional object recognition processing (S4) is implemented based on the position on the image and the detection area for detecting the three-dimensional object.

[0036] Figure 3This is a flowchart showing an example of a three-dimensional object lateral velocity calculation process which is the basis of the present invention. Figure 2 The processing is described by referring to the three-dimensional object detection processing (S3), various recognition processing (S4), and vehicle control processing (S5).

[0037] First, the 3D object detection unit 301 obtains an image obtained from a stereo camera or a monocular camera from the image data buffer 203 (S31). Next, the 3D object detection unit 301 detects 3D objects appearing in the image (S32). Next, the 3D object size calculation unit 302 calculates the 3D object size of the detected 3D object (S33).

[0038] Next, the 3D object recognition unit 303 performs recognition processing on the detected 3D object (S41) to determine whether the 3D object is a pedestrian. If the 3D object is a pedestrian, the lateral velocity calculation unit 304 calculates the pedestrian's lateral velocity (S42). The control processing unit 108 then performs vehicle control processing (S5), terminating the present process.

[0039] The lateral velocity calculated in step S42 is the velocity in the x-direction, with the front and rear axles of the host vehicle as the z-axis and the left and right axles of the host vehicle as the x-axis. It is the lateral component of the pedestrian's relative velocity with respect to the host vehicle. If a pedestrian is in the host vehicle's lane, or if the lateral velocity calculated above indicates that a pedestrian outside the host vehicle's lane is moving and intruding into the host vehicle's lane, an alarm or braking control is initiated in the vehicle control process (S16) to assist with the safety of the driver and pedestrian.

[0040] <Description of Processing for Calculating Pedestrian Lateral Velocity> Next, an example of a process of calculating the lateral velocity of a pedestrian using images obtained from a stereo camera will be described.

[0041] Figure 4 is shown based on the camera image, Figure 2 The result of the three-dimensional object detection process (S3) shown in FIG. Figure 2 The three-dimensional object detection process will be described in detail by taking the image 401 acquired by the stereo camera by the three-dimensional object detection unit 301 as an example.

[0042] When detecting a 3D object appearing in image 401, 3D object detection unit 301 sets multiple windows 402 and 403 of different sizes on image 401. Windows 402 and 403 are areas where the 3D object recognition unit 303 identifies whether a 3D object detected by the 3D object detection unit 301 at each position within image 401 is a pedestrian.

[0043] As mentioned above, through Figure 2 The parallax processing (S2) shown can determine the distance to the 3D object. Therefore, the 3D object detection unit 301 uses a large window 402 for 3D objects close to the vehicle and a small window 403 for 3D objects far from the vehicle, thereby accurately identifying 3D objects at different distances.

[0044] The 3D object detection unit 301 slides windows 402 and 403 within the image 401, performing 3D object recognition (scanning) at each position after sliding by a predetermined amount. When a 3D object is detected, rectangular detection areas 404 and 405 are displayed at the detected position within the image 401.

[0045] Detection areas 404 and 405, the detection results of the 3D object detection process ( S3 ), indicate the areas on the image where 3D objects such as pedestrians and vehicles existing in three-dimensional space are projected. The 3D object recognition unit 303 identifies whether a 3D object is a pedestrian based on detection areas 404 and 405 , thereby reducing processing costs.

[0046] Detection areas 404 and 405 can be rectangular or irregular shapes that change depending on parallax or distance. However, to facilitate computer processing in the 3D object recognition process (S4), they are generally treated as rectangular. Therefore, assuming that the 3D object recognition unit 303 uses a rectangular detection area 404 to recognize a 3D object, the following processing will be described in detail.

[0047] Next, conventional calculation processing and calculation results of the lateral velocity will be described. Figures 5A to 5C 1 is a diagram showing an example of conventional speed calculation processing and calculation results. Pedestrian recognition processing is performed based on a detection area 404 calculated from an image 401 obtained from a stereo camera.

[0048] Figure 5A 4 is a diagram showing an example of a right edge 501 of a pedestrian 500 appearing in the detection area 404. If the recognition result is a pedestrian 500, the process is transferred to Figure 3 The lateral velocity calculation process (S15) is shown. Conventional processing units refer to the left and right edges of pedestrian 500 to calculate the lateral velocity of pedestrian 500. The following description shows an example in which right edge 501 of pedestrian 500 is referenced. In the figure, right edge 501 is represented by a thick solid curve as the outline of the right side of pedestrian 500.

[0049] Figure 5BThis figure shows an example of average horizontal position 502 of right edge 501 calculated based on image 401 of frame t. For example, a conventional processing unit obtains the horizontal position of each pixel of right edge 501 and calculates the average horizontal position. This average horizontal position is represented by a thick solid line as average horizontal position 502. For example, the average horizontal position in detection area 404 is calculated based on the horizontal position coordinates of multiple pixels sampled multiple times from right edge 501.

[0050] Figure 5C 4 is a diagram showing an example of a horizontal position average value 502 of the right edge 501 calculated from the image 401 of the (t+1)th frame. Figure 5B The same process is performed on the t-th frame shown to obtain the average lateral position 502 of the right edge 501. Here, if the pedestrian 500 moves to the right, the movement amount 504 from the average lateral position 502 to the average lateral position 503 from the t-th frame to the (t+1)-th frame is calculated.

[0051] In this manner, the conventional processing unit calculates the average lateral position value 502 of the previous frame and the average lateral position value 502 of the current frame. The conventional processing unit then calculates the amount of movement 504 of the average lateral position value 502 for each frame, and calculates the lateral velocity of the pedestrian 500 based on the inter-frame time and the amount of movement 504. The greater the amount of movement 504, the faster the pedestrian 500 is moving in the direction of the movement of the average lateral position value 502.

[0052] Next, a description will be given of conventional problems caused by obtaining the average value of the horizontal position. Figures 6A to 6C This figure shows an example of an image captured while a pedestrian 600 is standing near a tall three-dimensional object such as a utility pole.

[0053] Figure 6A 601 is a diagram showing an example of a detection result when a non-target object 601 exists near a pedestrian 600. Figure 6A The image captured in is taken as the first frame. Figure 3 In the illustrated three-dimensional object detection process (S3), if a non-target object 601 other than the pedestrian being recognized, such as a utility pole or sign, is located near pedestrian 600, a detection area 602 larger than pedestrian 600 may be erroneously detected. In this case, since the edge position 603 of non-target object 601 is included in detection area 602, noise is superimposed on the edge position of pedestrian 600. Therefore, a lateral position average value 604, shown by the solid line, is calculated, which differs from the ideal lateral position average value 605, shown by the dotted line.

[0054] Figure 6B is a diagram showing an example of the detection result of the second frame. Figure 6B The image captured in the first frame is used as the second frame. Pedestrian 600 remains essentially stationary in both the first and second frames. However, suppose that, due to, for example, the vehicle moving forward, non-target object 601 no longer exists within the detection area in the second frame. In this case, only edge 606 for pedestrian 600 is calculated, and this edge 606 does not include the edge of non-target object 601. Therefore, the conventional processing unit calculates a lateral position average 607 for edge 606.

[0055] Figure 6C The figure shows an example of the amount of movement of the average lateral position. The amount of movement 608 of the average lateral position 607 calculated in the second frame is calculated to be excessively large relative to the average lateral position 604 calculated in the first frame. Due to this amount of movement 608, the lateral velocity of pedestrian 600 is incorrectly calculated to be excessively large even though the pedestrian is stationary. For example, one could imagine a situation where, while the vehicle is turning right or left, the pedestrian and a utility pole appear to overlap and be visible at one point, but then no longer overlap and be visible at the next point. Furthermore, one could imagine a situation where, while the pedestrian is far from the vehicle, the pedestrian and the utility pole appear to overlap and be visible, but as the vehicle approaches the pedestrian, the pedestrian and the utility pole no longer overlap and are visible.

[0056] As described above, in the conventional motion calculation process, a pedestrian 600 that is stationary and outside the vehicle's lane is mistakenly determined to be invading the vehicle's lane, leading to a problem of false alarms or false braking. Therefore, it is necessary to accurately calculate the pedestrian's motion even when a non-target object 601 is within the detection area.

[0057] Next, the movement amount calculation process according to the first embodiment will be described. Figure 7A and Figure 7B : is a diagram showing an example of movement amount calculation processing according to the first embodiment. Figure 7A and Figure 7B In the embodiment, a situation where the pedestrian 600 stands near a tall object such as a utility pole is also described.

[0058] The 3D object detection unit 301 of the vehicle-mounted stereo camera device 100 of the first embodiment first calculates the physical quantity of the 3D object size based on the detection area size and distance on the image. Here, since the size of the 3D object (especially pedestrians) detected by the 3D object detection unit 301 is unstable, even if the image Figure 6AAs shown, even if detection area 602 is detected to be larger than pedestrian 600, lateral velocity calculator 304 must also suppress erroneous velocity calculation of pedestrian 600. Therefore, two methods will be described in which lateral velocity calculator 304 uses the size of a three-dimensional object to set an edge sampling range for lateral velocity.

[0059] (The first method of setting the edge sampling range) First, refer to Figure 7A and 7B , an edge sampling method (first method) using a pre-assumed pedestrian size is described.

[0060] Figure 7A This is an example of a detection area 602 including a pedestrian 600 . Figure 7A An example in which the first frame image is acquired is shown in FIG.

[0061] Assume that the three-dimensional object detection unit 301 detects a detection area 602 that is larger than the pedestrian 600 because the pedestrian 600 is near a non-target object 601 (such as a utility pole). Therefore, the lateral velocity calculation unit 304 samples the edges within the area 701 excluding the portion exceeding the prescribed value from the road surface with reference to the physical quantity of the longitudinal width of the detection area 602 to calculate the lateral position average value 702. The prescribed value of the height from the road surface is set to, for example, 2.0 m. In addition, the prescribed value of the height from the road surface can also be set to, for example, 2.1 m, 2.2 m, etc. The prescribed value of the height from the road surface is set based on the height of a person. The movement information calculation unit (lateral velocity calculation unit 304) sets the sampling range by excluding the area in the image where the height from the road surface exceeds the prescribed value.

[0062] Figure 7B This is an enlarged view of the right side of the pedestrian 600 captured in the second frame. Figure 7B An example of acquiring the second frame image is shown in FIG.

[0063] There is no non-target object 601 in the detection area 602 of the second frame image. Figure 6B As described above, the lateral speed calculation unit 304 calculates the lateral position average value 607 using only the right edge of the pedestrian 600. As a result, the movement amount 703 from the lateral position average value 702 calculated in the first frame to the lateral position average value 607 calculated in the second frame is the same as that in the first frame. Figure 6C As described above, since the detected three-dimensional object is recognized as the pedestrian 600 from the multi-frame image, the lateral speed calculation unit 304 can accurately calculate the lateral speed of the pedestrian 600.

[0064] (The second method for setting the edge sampling range) Next, refer to Figure 8A and Figure 8B , the edge sampling method (second method) using the recognition results is explained.

[0065] Figure 8A 1 is a diagram showing an example of a detection area 710 including a pedestrian 600 . Figure 8A An example in which the first frame image is acquired is shown in FIG.

[0066] The 3D object recognition unit 303 associates the recognition result of the 3D object recognized in the first frame with the 3D object size calculated by the 3D object size calculation unit 302 and stores the result in the storage unit 106 for reference in the processing of the next frame. Figure 3 In the three-dimensional object recognition process (S41), the recognition score of the detection area 710 is calculated. For example, when the minimum value of the recognition score is set to 0 and the maximum value is set to 100, as shown in FIG. Figure 8A As shown, if a detection area 710 that contains a pedestrian 600 to be recognized at a large ratio is acquired, a high recognition score is assigned to the detection area 710 .

[0067] If the detection process of the object to be recognized by the 3D object detection unit 301 is stable, the recognition accuracy of the object to be recognized by the 3D object recognition unit 303 will be high. For example, if the recognition score calculated for the detection area 710 in units of frames is high, the detection process of the object to be recognized is stable. Therefore, it can be inferred that the 3D object size calculated by the 3D object size calculation unit 302 based on the detection area 710 is also stable. Assuming that Figure 8A In the first frame of the image shown, the recognition score of the pedestrian 600 is very high due to the stable detection area 710.

[0068] Figure 8B 1 is a diagram showing an example in which a detection area 602 larger than the recognition target is acquired. Figure 8B An example of acquiring the second frame image is shown in FIG.

[0069] For example, suppose that after the 3D object detection unit 301 acquires the first frame of image, the setting of the detection area becomes unstable, and in the next frame the 3D object detection unit 301 acquires a detection area 602 that is larger than the recognition target (pedestrian 600). In this case, the upper side of the detection area 602 includes the edge position 603 of the non-target object 601, and the blank area outside the pedestrian 600 is larger than the detection area 602. Figure 8A The detection area 710 shown is larger. Therefore, the lateral speed calculation unit 304 cannot calculate an ideal lateral position average value as the edge of the pedestrian 600.

[0070] Therefore, the movement information calculation unit (lateral velocity calculation unit 304) sets the sampling range based on the recognition result indicating that the three-dimensional object is the recognition target and the size of the three-dimensional object calculated in the image of the frame in which the recognition result is obtained. Figure 8A The 3D object size 704 in the detection area 710 of the first frame, determined to have a high recognition score in the image, is set as the edge sampling range for calculating the lateral velocity in the next frame, the second frame. By calculating the lateral position average value 705 within the sampling range set based on the 3D object size 704, the lateral velocity calculation unit 304 can prevent the movement of the pedestrian 600 from being calculated to be greater than the actual movement.

[0071] In the second method, since the recognition score is used, for example, even if the detection processing becomes unstable and a large detection area 602 is obtained in the image of the second frame, the edge of the three-dimensional object can be sampled using the sampling range set based on the detection area 710 with a high recognition score in the image of the first frame.

[0072] Furthermore, the second method does not set a predetermined distance from the road surface, as described in the first method. Therefore, the height of detection area 710, which has a high recognition score, is variable, sometimes 1.5 meters and sometimes 1.8 meters. However, since this method does not require a predetermined distance from the road surface, it offers greater versatility than the first method.

[0073] By using the first method or the second method described above, the lateral speed calculation unit 304 can suppress an increase in the amount of movement 703 of the lateral position average value, thereby preventing erroneous speed calculation of the stationary pedestrian 600 .

[0074] Figure 9 yes Figure 2 A flowchart of an example of detailed processing of the three-dimensional object recognition processing (S4).

[0075] First, the three-dimensional object recognition unit 303 Figure 3 In the illustrated 3D object recognition process (S41), pedestrian recognition processing (S51) is performed for the detection area obtained in the 3D object detection process (S3). In the pedestrian recognition process, the 3D object recognition unit 303 determines whether the 3D object detected in the detection area is a pedestrian based on the recognition results of multiple frames (S52). If the 3D object recognition unit 303 determines that the object to be recognized is not a pedestrian ("No" in S52), this process ends.

[0076] On the other hand, if the three-dimensional object recognition unit 303 determines that the recognition target is a pedestrian (Yes in S52 ), the process moves to the lateral speed calculation process ( S42 ). In the lateral speed calculation process ( S42 ), the lateral speed calculation unit 304 calculates the left and right edges of the pedestrian ( S61 ).

[0077] Next, the lateral velocity calculation unit 304 calculates the 3D object size of the pedestrian as a 3D object (S62). In the 3D object size calculation process, the lateral velocity calculation unit 304 calculates the physical quantity of the 3D object size (e.g., height) based on the detection area and distance on the image. Note that the 3D object size calculated in step S62 is the size of the pedestrian, and Figure 2 The 3D object size calculated by the 3D object size calculation unit 302 shown in FIG is the size of the 3D object for which it is not yet clear whether it is a pedestrian. In addition, the lateral speed calculation unit 304 may obtain the 3D object size from the 3D object size calculation unit 302 in step S62 instead of calculating the 3D object size. Figure 3 The size of the three-dimensional object calculated in step S33.

[0078] Next, the lateral velocity calculation unit 304 samples the left and right edges of the 3D object using the 3D object size calculated in step S62 ( S63 ) Next, the lateral velocity calculation unit 304 calculates the average lateral position of the sampled left and right edges ( S64 ).

[0079] Next, the lateral velocity calculation unit 304 calculates the amount of movement of the average lateral position of the left and right edges in the current frame based on the average lateral position of the left and right edges calculated in the previous frame and the average lateral position of the left and right edges in the current frame (S65). Finally, the lateral velocity calculation unit 304 calculates the lateral velocity of the pedestrian (S66), and this process ends.

[0080] <Computer Hardware Configuration Example> Next, the hardware configuration of the computer 800 constituting each device of the vehicle-mounted stereo camera device 100 will be described.

[0081] Figure 10 This is a block diagram showing an example of the hardware configuration of a computer 800. The computer 800 is an example of hardware that can be used as a computer that can operate as the vehicle-mounted stereo camera device 100 of this embodiment. The vehicle-mounted stereo camera device 100 of this embodiment is implemented by the computer 800 (computer) executing a program. Figure 1 and Figure 2 The functional blocks shown cooperate to perform various processes.

[0082] The computer 800 includes a CPU (Central Processing Unit) 801 , a ROM (Read Only Memory) 802 , and a RAM (Random Access Memory) 803 , each of which is connected to a bus 804 . Furthermore, the computer 800 includes a nonvolatile memory 805 and a network interface 806 .

[0083] The CPU 801 reads the program code of the software that implements the various functions of this embodiment from the ROM 802, loads it into the RAM 803, and executes it. Variables and parameters generated during the CPU 801's calculations are temporarily written to the RAM 803, and these variables and parameters are read out as appropriate by the CPU 801. However, an MPU (Micro Processing Unit) or a GPU (Graphics Processing Unit) may be used in place of the CPU 801, or both the CPU 801 and the GPU (Graphics Processing Unit) may be used.

[0084] Nonvolatile memory 805 may include, for example, a hard disk drive (HDD), a solid state drive (SSD), a floppy disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, or a nonvolatile memory. The nonvolatile memory 805 stores an operating system (OS), various parameters, and programs for operating the computer 800. The ROM 802 and nonvolatile memory 805 store programs and data required for the operation of the CPU 801 and serve as an example of a computer-readable, non-transitory storage medium that stores programs executed by the computer 800.

[0085] As the network interface 806 , for example, a NIC (Network Interface Card) or the like is used, and various data can be transmitted and received between devices via a LAN (Local Area Network) connected to a terminal of the NIC, a dedicated line, or the like.

[0086] The vehicle-mounted stereo camera device 100 of the first embodiment described above, when using a stereo camera as a sensor for detecting external information, calculates parallax information from images captured by the stereo camera to obtain images of three-dimensional objects. Subsequently, when the vehicle-mounted stereo camera device 100 identifies the three-dimensional object identified in the image as a pedestrian, it calculates the left and right edges of the pedestrian and calculates the size of the three-dimensional object based on the left and right edges. Here, when the three-dimensional object is identified as a pedestrian, the sampling range is set based on the size of the three-dimensional object so that the edges of non-objects adjacent to the pedestrian in the image are not sampled. As a result, even when the detection of the three-dimensional object being identified is unstable, it is possible to calculate the lateral velocity with high accuracy. Furthermore, by improving the accuracy of the lateral velocity calculation, the accuracy of the warning or braking control used by the vehicle to avoid collision with the three-dimensional object is improved, thereby contributing to safety assistance for the driver and pedestrians.

[0087] Furthermore, by identifying whether a 3D object is a recognition target based on images captured over multiple frames, the 3D object recognition unit 303 can accurately identify the 3D object as a recognition target even if the 3D object appears near a non-target object in an image of a previous frame, but appears separated from the non-target object in an image of a subsequent frame.

[0088] In the first method, the sampling range is set by setting the height from the road surface of the pedestrians appearing in the image to the average height of the pedestrians being identified, and excluding areas exceeding this height from the road surface. This prevents the lateral velocity calculation unit 304 from sampling non-target objects that are taller than the identification targets, and thus preventing the lateral position values ​​calculated based on these non-target objects from being included in the average lateral position value.

[0089] In the second method, the recognition result indicating that a 3D object is a target for recognition is expressed as a recognition score. It is understood that higher recognition scores indicate more stable detection of the target. Consequently, the 3D object size calculated based on the detection area with a high recognition score also becomes more stable, allowing the sampling range to be set based on this 3D object size.

[0090] Furthermore, the vehicle-mounted stereo camera device 100 of the present embodiment can accurately calculate the movement of a pedestrian even when the pedestrian is walking in a place other than a crosswalk.

[0091] Furthermore, in the above embodiment, although pedestrians are identified as the target, at least one of cyclists, motorcyclists, and people riding personal mobility vehicles may also be identified as the target in addition to pedestrians.

[0092] Furthermore, while the above embodiment illustrates an example in which lateral velocity calculation unit 304 uniformly samples the left and right edges of pedestrian 600, it is also possible to weight the sampled information. For example, when pedestrian 600 moves, the lower body moves more than the upper body, so the movement of the lower body can easily cause errors in the average lateral position. Therefore, by assigning a higher weight to the sampled information of pedestrian 600's upper body than to the lower body, lateral velocity calculation unit 304 can accurately determine the average lateral position of pedestrian 600.

[0093] [Second embodiment] Next, refer to Figure 11 The following describes a configuration example of a vehicle-mounted camera device according to a second embodiment of the present invention. The vehicle-mounted camera device according to the second embodiment uses a monocular camera and a millimeter-wave radar as sensors, instead of the stereo camera used in the first embodiment.

[0094] Figure 11 2 is a block diagram showing a configuration example of the vehicle-mounted camera device 900 . The vehicle-mounted camera device 900 includes a monocular camera 901, a millimeter-wave radar 902, an image data buffer 903, a distance measurement data buffer 904, and a CAN IF 910. Figure 1 Therefore, the monocular camera 901 is connected to the image input IF 103 (refer to Figure 1 ), image data input through the image input IF 103 is stored in the storage unit 106 via the internal bus 109. In addition, an input IF (not shown) for the millimeter-wave radar 902 is provided, and data output by the millimeter-wave radar 902 is input through this input IF and stored in the storage unit 106 via the internal bus 109.

[0095] The monocular camera (monocular camera 901) is an example of a camera unit having one imaging element, and captures the environment in front of the vehicle. The image data of the image captured by the monocular camera 901 is processed by the image processing unit 104 (S21). The content of the image processing (S21) is the same as Figure 2 The image data after image processing is buffered in the image data buffer 903 provided in the storage unit 106. In addition, unlike the first embodiment, since the camera unit is not configured as a stereo camera, the vehicle-mounted camera device 900 of the second embodiment does not perform the image processing (S1). Figure 2 The parallax processing (S2) is shown.

[0096] The distance measurement unit (millimeter-wave radar 902) measures the distance to three-dimensional objects. For example, millimeter-wave radar 902 radiates millimeter-wave radio waves in front of the vehicle and measures the distance to the object that reflects the radio waves, the object's speed, and other parameters. The data output by millimeter-wave radar 902 undergoes distance measurement processing in the arithmetic processing unit 105 (S22). The processed distance measurement data is buffered in the distance measurement data buffer 904 provided in the storage unit 106.

[0097] After that, the three-dimensional object detection process (S23) is performed using the image data read out from the image data buffer 903 and the distance measurement data read out from the distance measurement data buffer 904. The three-dimensional object detection process (S23) corresponds to Figure 2 The three-dimensional object detection process (S3) shown in the figure switches to use according to the distance measurement data Figure 4 The detection areas 404 and 405 shown are used to detect three-dimensional objects.

[0098] The three-dimensional object recognition process (S24) and the vehicle control process (S25) after the three-dimensional object detection process (S23) correspond to Figure 2 and Figure 3 The three-dimensional object recognition process (S4) and the vehicle control process (S5) shown in FIG. For example, the three-dimensional object detection unit (three-dimensional object detection unit 301) detects a three-dimensional object at a specific distance based on the image input from the camera element. The three-dimensional object recognition unit (three-dimensional object recognition unit 303) recognizes the detected three-dimensional object. In addition, Figure 3 In the three-dimensional object size calculation process ( S33 ) of the three-dimensional object detection process ( S3 ) shown, the three-dimensional object size calculation unit 302 can calculate the three-dimensional object size based on the image obtained from the monocular camera 901 and the distance and detection size obtained by the millimeter wave radar 902 .

[0099] CAN IF 910 transmits information output from the vehicle-mounted camera device 900 to the Figure 1 The in-vehicle network CAN 110 shown is output to a control system (not shown) of the host vehicle.

[0100] Furthermore, the present invention can also be applied to a device including a sensor capable of simultaneously measuring an image and a distance, in addition to the monocular camera 901 and the millimeter-wave radar 902 .

[0101] In the vehicle-mounted camera device 900 of the second embodiment described above, similar to the vehicle-mounted stereo camera device 100 of the first embodiment, when the three-dimensional object detected in the image is recognized as a pedestrian to be recognized, the lateral velocity can be calculated using the first and second methods described above. Therefore, even if there are non-target objects such as tall utility poles near the pedestrian, the pedestrian's lateral velocity can be accurately calculated.

[0102] The present invention is not limited to the above-described embodiments, and various other application examples and modified examples are of course possible without departing from the gist of the present invention described in the claims. For example, the above embodiments provide detailed and specific descriptions of the configuration of the environment recognition device to facilitate understanding of the present invention. These descriptions are not necessarily limited to devices having all of the configurations described. Furthermore, a portion of the configuration of the embodiments described herein may be replaced with a configuration of another embodiment, and a configuration of another embodiment may be added to a configuration of a particular embodiment. Furthermore, other configurations may be added, deleted, or substituted for a portion of the configuration of each embodiment. Furthermore, the control lines and information lines shown are those necessary for the purpose of explanation and do not necessarily represent all the control lines and information lines on the product. In practice, it is assumed that almost all components are interconnected. Explanation of symbols

[0103] 100…In-vehicle stereo camera device, 101…Left camera, 102…Right camera, 103…Image input IF, 104…Image processing unit, 105…Calculation processing unit, 106…Storage unit, 108…Control processing unit, 109…Internal bus, 110…In-vehicle network CAN, 203…Image data buffer, 301…Three-dimensional object detection unit, 302…Three-dimensional object size calculation unit, 303…Three-dimensional object recognition unit, 304…Lateral velocity calculation unit.

Claims

1. An environment recognition device, characterized in that: have: a three-dimensional object detection unit that sets a detection area for detecting an identification object in the image captured by the camera unit and detects a three-dimensional object appearing in the image based on the detection area; a three-dimensional object size calculation unit for calculating the size of the three-dimensional object detected in the detection area using information on the distance from the device to the three-dimensional object calculated based on the image; a three-dimensional object recognition unit for recognizing that the three-dimensional object detected in the detection area is a recognition target; as well as A movement information calculation unit limits a sampling range of the three-dimensional object based on a size of the three-dimensional object and calculates movement information of a lateral movement of the three-dimensional object as the recognition target based on edge information of the three-dimensional object as the recognition target.

2. The environment recognition device according to claim 1, characterized in that The three-dimensional object recognition unit recognizes that the three-dimensional object is a recognition target based on the continuous plurality of frames of the image. The movement information calculation unit calculates an average value of horizontal coordinates of points sampling edges of the three-dimensional object frame by frame as the edge information, and calculates a lateral speed of the three-dimensional object to be recognized as the movement information.

3. The environment recognition device according to claim 2, characterized in that: The movement information calculation unit sets the sampling range by excluding a region whose height from the road surface appearing in the image exceeds a predetermined value.

4. The environment recognition device according to claim 2, characterized in that: The movement information calculation unit sets the sampling range based on a recognition result indicating that the three-dimensional object is the recognition target and a size of the three-dimensional object calculated in the image of the frame in which the recognition result is obtained.

5. The environment recognition device according to claim 3, characterized in that: The movement information calculation unit samples an edge on either the left or right side of the three-dimensional object and calculates an average value of horizontal coordinates of points where the edge is sampled.

6. The environment recognition device according to claim 5, characterized in that: The camera unit is a stereo camera having two imaging elements. The environment recognition device includes an image processing unit that calculates the parallax of the three-dimensional object based on the images input from the respective imaging elements. The three-dimensional object detection unit detects the three-dimensional object at the same distance from the device and the vehicle equipped with the stereo camera. The three-dimensional object recognition unit recognizes the three-dimensional objects at the same distance.

7. The environment recognition device according to claim 5, characterized in that: The camera unit is a monocular camera having one imaging element. The environment recognition device includes a distance measuring unit that measures the distance to the three-dimensional object. The three-dimensional object detection unit detects the three-dimensional object at a specific distance based on the image input from the imaging element. The three-dimensional object recognition unit recognizes the detected three-dimensional object.

8. The environment recognition device according to claim 5, characterized in that: The three-dimensional object as the recognition target is at least any one of a pedestrian, a cyclist, a motorcyclist, and a person riding a personal transportation vehicle.

9. The environment recognition device according to claim 5, characterized in that: The lateral direction is the direction in which a pedestrian crosses the front of a vehicle equipped with the present device or the lane in which the vehicle is traveling.

10. An environment recognition method, characterized in that: include: a step of setting a detection area for detecting an identification object in an image captured by a camera unit, and detecting a three-dimensional object appearing in the image based on the detection area; a step of calculating the size of the three-dimensional object detected in the detection area using information on the distance from the device to the three-dimensional object calculated based on the image; a step of identifying that the three-dimensional object detected in the detection area is an identification target; as well as The step of limiting a sampling range of the three-dimensional object based on a size of the three-dimensional object and calculating movement information of a lateral movement of the three-dimensional object as the recognition target based on edge information of the three-dimensional object as the recognition target.

Images