Environment recognition device and environment recognition method

The environment recognition device addresses the challenge of accurately detecting pedestrians and calculating their lateral speed by using a three-dimensional object detection and movement information calculation system, enhancing the reliability of collision avoidance systems and reducing false alarms.

WO2024247216A1PCT designated stage expired Publication Date: 2024-12-05ASTEMO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2023/020457
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-06-01
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing in-vehicle collision safety systems face challenges in accurately detecting pedestrians and calculating their lateral speed, especially under conditions like night-time imaging or when background objects interfere, leading to incorrect edge extraction and potential false braking warnings.

Method used

An environment recognition device with a three-dimensional object detection unit, size calculation unit, identification unit, and movement information calculation unit, which sets a detection area, calculates object size based on distance, and limits sampling range to accurately determine pedestrian movement, even in unstable detection scenarios.

Benefits of technology

This solution enables accurate calculation of pedestrian movement information, reducing false alarms and improving the reliability of collision avoidance systems by enhancing the accuracy of lateral speed calculation and vehicle control processes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023020457_05122024_PF_FP_ABST
    Figure JP2023020457_05122024_PF_FP_ABST
Patent Text Reader

Abstract

An environment recognition device according to the present invention includes: a three-dimensional object detection unit that sets a detection region for detecting an object to be identified with respect to an image captured by an image-capturing unit, and detects a three-dimensional object captured in the image on the basis of the detection region; a three-dimensional object size calculation unit that calculates the size of the three-dimensional object detected in the detection region using information regarding a distance from an own device to the three-dimensional object that is calculated from the image; a three-dimensional object identification unit that identifies that the three-dimensional object detected in the detection region is the object to be identified; and a lateral speed calculation unit that limits a sampling range of the three-dimensional object on the basis of the size of the three-dimensional object, and calculates movement information in which the three-dimensional object to be identified moves in a lateral direction on the basis of information regarding an edge of the three-dimensional object to be identified.
Need to check novelty before this filing date? Find Prior Art

Description

Environment recognition device and environment recognition method

[0001] The present invention relates to an environment recognition device and an environment recognition method.

[0002] In recent years, the widespread use of in-vehicle sensing devices has led to a demand for improved performance in collision safety features for pedestrians. The New Car Assessment Programme (NCAP) was established as a vehicle safety assessment. For this reason, Euro NCAP, Japan NCAP, and other standards are used in different regions. Various assessments are being actively added or changed to the vehicle assessment. As the NCAP continues to evolve, it is necessary to keep up with improvements in the performance of collision safety features. One example of the performance assessment of collision safety features is the evaluation of automatic braking for pedestrians running out into the road.

[0003] In order for the ECU installed in the vehicle to determine whether a pedestrian has jumped out into the road, the ECU first identifies the pedestrian from an image captured by the onboard camera and calculates the lateral movement speed of the pedestrian. Next, if the ECU determines that the pedestrian is moving upward in the vehicle's lane based on the movement speed and direction, it activates Autonomous Emergency Braking (AEB).

[0004] Patent Document 1 states that "a contour line is generated in which the peak points of the edges continue upward in the image, and the contour lines are classified into groups based on the horizontal movement of the contour lines between captured images. Based on the horizontal movement of the contour line group and the height size of the contour line group, it is accurately determined whether or not there is a pedestrian who is at high risk of jumping into the path of the vehicle."

[0005] JP 2012-203884 A

[0006] Conventionally, the three-dimensional object identification process performed by an ECU (Electronic Control Unit) is based on the assumption that three-dimensional objects are accurately detected, and by maintaining the identification performance when identifying the detection area of ​​the three-dimensional object, processing costs are reduced. However, in reality, three-dimensional objects may not be accurately detected due to factors such as nighttime, the color of clothing and the background, distance, etc. For example, if there is a utility pole or signpost near a pedestrian, an area larger than the actual size of the pedestrian may be detected.

[0007] One method for calculating a pedestrian's lateral speed uses the pedestrian's edge within the detection area. The technology described in Patent Document 1 extracts edges (contour lines) from an image and determines whether a three-dimensional object detected in the detection area is a pedestrian based on the size of the edges. If the determination result is a pedestrian, the pedestrian's lateral speed is calculated. If the determination result is not a pedestrian, the object is treated as a non-pedestrian. However, the technology described in Patent Document 1 cannot accurately extract edges depending on the pedestrian's background. As a result, there is a risk that the edge will be treated as a non-pedestrian and the lateral speed will not be calculated accurately. Furthermore, as described above, when a region larger than the actual size of a pedestrian is detected, the method using edges within the detection area may incorrectly calculate the lateral speed of a stopped pedestrian. Such an incorrect lateral speed calculation poses a problem in the ECU's vehicle control process, which may issue an alarm or perform sudden braking even in situations where there is no risk of a pedestrian jumping out.

[0008] The present invention has been made in view of the above circumstances, and has as its object to accurately calculate movement information of a three-dimensional object to be identified.

[0009] The environmental recognition device of the present invention comprises a three-dimensional object detection unit that sets a detection area for detecting identification targets in an image captured by an imaging unit and detects three-dimensional objects that appear in the image based on the detection area; a three-dimensional object size calculation unit that calculates the size of the three-dimensional object detected in the detection area using information about the distance from the device to the three-dimensional object calculated from the image; a three-dimensional object identification unit that identifies the three-dimensional object detected in the detection area as the identification target; and a movement information calculation unit that limits the sampling range of the three-dimensional object based on the size of the three-dimensional object and calculates movement information of the three-dimensional object that moves laterally based on information about the edges of the three-dimensional object that is the identification target.

[0010] According to the present invention, by limiting the sampling range of a three-dimensional object based on the size of the three-dimensional object, it is possible to accurately calculate movement information of the three-dimensional object to be identified even when detection of the three-dimensional object is unstable.

[0011] 1 is a block diagram showing an example of the overall configuration of an in-vehicle stereo camera device according to a first embodiment of the present invention. FIG. 1 is a flowchart and configuration diagram showing an example of processing performed by each functional unit of the in-vehicle stereo camera device according to the first embodiment of the present invention. FIG. 2 is a flowchart showing an example of processing for calculating the lateral speed of a three-dimensional object, which is the basis of the present invention. FIG. 3 is a diagram showing the results of a three-dimensional object detected by three-dimensional object detection processing based on camera images according to the first embodiment of the present invention. FIG. 4 is a diagram showing an example of the right edge of a pedestrian appearing in a detection area. FIG. 5 is a diagram showing an example of the average lateral position of the right edge calculated from the image of the tth frame. FIG. 6 is a diagram showing an example of the average lateral position of the right edge calculated from the image of the (t+1)th frame. FIG. 7 is a diagram showing an example of detection results when a non-target object is present near a pedestrian. FIG. 8 is a diagram showing an example of the detection result of a second frame. FIG. 9 is a diagram showing an example of an amount of movement of the average lateral position. FIG. 10 is an example of a detection area including a pedestrian. FIG. 11 is an enlarged view of the right part of a pedestrian captured in the second frame. FIG. 12 is a diagram showing an example of a detection area including a pedestrian. FIG. 13 is a diagram showing an example in which a detection area larger than the target is obtained. FIG. 14 is a flowchart showing an example of detailed processing of three-dimensional object identification processing according to the first embodiment of the present invention. FIG. 15 is a block diagram showing an example of the hardware configuration of a computer according to the first embodiment of the present invention. FIG. 16 is a block diagram showing an example of the configuration of an in-vehicle camera device according to a second embodiment of the present invention.

[0012] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functions or configurations are designated by the same reference numerals, and redundant description will be omitted. The present invention is applicable to, for example, a computing device for vehicle control capable of communicating with an on-board ECU (Electronic Control Unit) for an Advanced Driver Assistance System (ADAS) or Autonomous Driving (AD).

[0013] First Embodiment FIG. 1 is a block diagram showing an example of the overall configuration of an in-vehicle stereo camera device 100 according to a first embodiment of the present invention.

[0014] The in-vehicle stereo camera device 100 is an example of an environment recognition device mounted on a vehicle (not shown) that recognizes the external environment based on image information captured from a target area in front of the vehicle, i.e., the vehicle's traveling direction. Based on the image information, the in-vehicle stereo camera device 100 identifies, for example, road markings, pedestrians, vehicles, other three-dimensional objects, traffic signals, signs, and illuminating lamps. Furthermore, the in-vehicle stereo camera device 100 adjusts the brakes, steering, and other aspects of the vehicle (hereinafter referred to as the "host vehicle") equipped with the in-vehicle stereo camera device 100 based on the results of identifying the external environment. The in-vehicle stereo camera device 100 safely controls the host vehicle by calculating movement information (movement speed, movement distance, or movement direction) of pedestrians moving laterally. In the following description, the direction in which a pedestrian moves in front of the vehicle (in-vehicle stereo camera device 100) equipped with the device or across the host vehicle's lane is referred to as the "lateral direction." The lateral direction may be a direction perpendicular to or oblique to the traveling direction of the vehicle or the direction of the lane in which the vehicle is traveling.

[0015] The in-vehicle stereo camera device 100 includes a left camera 101, a right camera 102, an image input I / F (Interface) 103, an image processing unit 104, an arithmetic processing unit 105, a storage unit 106, a CAN (Controller Area Network) IF 107, a control processing unit 108, and an internal bus 109.

[0016] The left camera 101 and the right camera 102 are disposed on the left and right sides of the vehicle, respectively, and capture images of the environment ahead of the vehicle using the imaging elements mounted on each camera to acquire image information. In the following description, the left camera 101 and the right camera 102 are also referred to as "stereo cameras" and "imaging units." Furthermore, image information output by the imaging elements of the stereo cameras is abbreviated as "image." The imaging units (left camera 101, right camera 102) are stereo cameras having two imaging elements. In FIG. 1 , the stereo camera is configured with two cameras, the left camera 101 and the right camera 102, each equipped with one imaging element. However, even if one camera is configured with two imaging elements aligned horizontally, the camera may be treated as a stereo camera.

[0017] The image input IF 103 is connected to the imaging elements of the stereo camera. The image input IF 103 controls the imaging operation of the stereo camera and captures images captured by the stereo camera. The image data captured through the image input IF 103 is sent to each unit via an internal bus 109. The image processing unit 104 and the arithmetic processing unit 105 then perform predetermined processing on the image data, and intermediate processing results, image data serving as the final result, and the like are stored in a storage unit 106.

[0018] The image processing unit 104 compares a first image obtained from the imaging element of the left camera 101 with a second image obtained from the imaging element of the right camera 102, and performs image corrections on each image, such as correcting for device-specific deviations due to the imaging elements and noise interpolation. The image processing unit 104 also stores the corrected image in the storage unit 106. Furthermore, the image processing unit (image processing unit 104) calculates the parallax of a three-dimensional object based on the first image and the second image input from the left and right imaging elements. At this time, the image processing unit 104 calculates corresponding points between the first and second images to calculate parallax information. The image processing unit 104 then stores the parallax information in the storage unit 106, similar to the process used to store the corrected image in the storage unit 106. In the following description, when the first image and the second image are not distinguished, they will be abbreviated as "image."

[0019] The arithmetic processing unit 105 performs an identification process for various three-dimensional objects necessary for identifying the environment around the vehicle, using the images (images captured by the left camera 101 or the right camera 102) stored in the memory unit 106 and parallax information (distance information between each point on the image). Examples of the various three-dimensional objects include people, cars, other obstacles, traffic lights, signs, and car taillights or headlights. The identification results by the arithmetic processing unit 105 and some of the intermediate calculation results are stored in the memory unit 106. The control processing unit 108 calculates vehicle control using the image identification results stored in the memory unit 106.

[0020] The control processing unit 108 calculates a vehicle control policy using the image and the identification results of various three-dimensional objects by the calculation processing unit 105. The vehicle control policy obtained as a result of the calculation by the control processing unit 108 and part of the identification results of various three-dimensional objects by the calculation processing unit 105 are transmitted to the in-vehicle network CAN 110 via the CAN IF 107, and the vehicle is braked by an accelerator or brake (not shown). Furthermore, the control processing unit 108 monitors these operations to see if each processing unit is operating abnormally or if an error has occurred during data transfer, thereby preventing abnormal operations.

[0021] The image processing unit 104 is connected to the image input IF 103, the arithmetic processing unit 105, the storage unit 106, and the control processing unit 108 via an internal bus 109. The image processing unit 104 is also connected to a CAN IF 107 of an in-vehicle network CAN 110 that is external to the in-vehicle stereo camera device 100 via the internal bus 109. As will be described later, the control processing unit 108, the image processing unit 104, the arithmetic processing unit 105, the storage unit 106, and the CAN IF 107 are configured by a single or multiple computer units.

[0022] The storage unit 106 is configured by a memory or the like that stores, for example, images processed by the image processing unit 104 and scan results of images scanned by the arithmetic processing unit 105. The CAN IF 107 performs input and output with an external in-vehicle network. For this purpose, the CAN IF 107 outputs information output from the in-vehicle stereo camera device 100 to a control system (not shown) of the vehicle via the in-vehicle network CAN 110.

[0023] Fig. 2 is a flowchart and a configuration diagram showing an example of processing performed by each functional unit of the in-vehicle stereo camera device 100. Fig. 2 shows image recognition processing as an example of an environment recognition method performed by the in-vehicle stereo camera device 100. Each processing shown in Fig. 2 will be explained by associating it with the reference numerals of each functional unit shown in Fig. 1.

[0024] First, images are captured by the stereo cameras (left camera 101, right camera 102) of the in-vehicle stereo camera device 100. The image processing unit 104 performs image processing (S1) such as correction to absorb peculiar characteristics of the imaging elements of the stereo cameras on each of the image data 201 captured by the left camera 101 and the image data 202 captured by the right camera 102. The results of the image processing are stored in the image data buffer 203. The image data buffer 203 is provided in the storage unit 106 of FIG. 1 and has a function of temporarily storing data related to the images.

[0025] Next, the image processing unit 104 compares the two corrected images read from the image data buffer 203. At this time, the image processing unit 104 performs parallax processing (S2) to obtain parallax information between the images (called "left and right images") output by the imaging elements of the stereo camera by comparing the two images. The parallax information between the left and right images makes it clear which points of interest on a three-dimensional object correspond to which points on the stereo camera images, and the distance to the object can be obtained by the principle of triangulation.

[0026] As described above, the image processing (S1) and the parallax processing (S2) are performed by the image processing unit 104 in Figure 1, and the final processed image, parallax information, and distance to the object are stored in the memory unit 106.

[0027] Next, the three-dimensional object detection unit 301 of the calculation processing unit 105 performs a three-dimensional object detection process (S3) to detect a three-dimensional object in three-dimensional space using the parallax information obtained in the parallax process (S2). In the three-dimensional object detection process (S3), the three-dimensional object detection unit (three-dimensional object detection unit 301) detects three-dimensional objects that are at the same distance from the device itself and from the vehicle on which the stereo camera is mounted. The three-dimensional object detection process (S3) includes an image acquisition process (S31) and a three-dimensional object detection process (S32) by the three-dimensional object detection unit 301, as well as a process of calculating the size of the three-dimensional object by the three-dimensional object size calculation unit 302, as shown in FIG. 3 (described later).

[0028] In the three-dimensional object detection process (S3), the three-dimensional object detection unit (three-dimensional object detection unit 301) acquires an image captured by the imaging unit (left camera 101, right camera 102), sets a detection area for detecting an identification target in this image, and detects three-dimensional objects appearing in the image based on the detection area (S31, S32 in FIG. 3). The three-dimensional object size calculation unit (three-dimensional object size calculation unit 302) calculates the size of the three-dimensional object detected in the detection area using information about the distance from the device to the three-dimensional object calculated from the image (S33 in FIG. 3).

[0029] Next, the three-dimensional object identification unit 303 of the arithmetic processing unit 105 performs a three-dimensional object identification process (S4) using the image, parallax information, and distance to the object stored in the memory unit 106. Three-dimensional objects to be identified in the three-dimensional object identification process (S4) include people, cars, other three-dimensional objects, signs, traffic lights, taillights, etc. The details of the three-dimensional object identification process (S4) are determined based on constraints such as the characteristics of the object and the processing time available in the system. The three-dimensional object identification process (S4) includes a three-dimensional object identification process (S41) by the three-dimensional object identification unit 303 shown in FIG. 3 (described later) as well as a lateral velocity calculation process (S42) by the lateral velocity calculation unit 304.

[0030] In the three-dimensional object identification process (S4), the three-dimensional object identification unit (three-dimensional object identification unit 303) identifies a three-dimensional object detected in the detection area as an identification target (S41). A three-dimensional object identified as an identification target by the three-dimensional object identification unit (three-dimensional object identification unit 303) has the same distance in two images. The three-dimensional object identification unit (three-dimensional object identification unit 303) identifies a three-dimensional object as an identification target based on images of multiple consecutive frames. The three-dimensional object identification unit 303 can identify a three-dimensional object as a pedestrian, which is an identification target, by pattern matching of the posture of a person when walking or the shape of a person, etc.

[0031] The movement information calculation unit (lateral velocity calculation unit 304) limits the sampling range of the three-dimensional object (pedestrian) based on the size of the three-dimensional object, and calculates movement information (lateral velocity) of the three-dimensional object to be identified moving laterally based on edge information of the three-dimensional object to be identified (S42). Here, the movement information calculation unit (lateral velocity calculation unit 304) calculates, for each frame, the average value of the horizontal coordinates of the sampled points of the three-dimensional object's edge as edge information, and calculates, as movement information, the lateral velocity of the three-dimensional object to be identified.

[0032] The movement information calculation unit (lateral velocity calculation unit 304) samples either the left or right edge of the three-dimensional object (pedestrian) and calculates the average value of the horizontal coordinates of the sampled edge points. For example, the lateral velocity calculation unit 304 samples edges of the three-dimensional object in a range from the side closest to the road surface (e.g., the toes or heels) to the side furthest from the road surface (e.g., the head or hat). If the object to be identified is a motorcycle, the edges are sampled in a range from the motorcycle wheels closer to the road surface to the helmet furthest from the road surface. Furthermore, in countries where people drive on the left side of the road, the right edge of the pedestrian is sampled, and in countries where people drive on the right side of the road, the left edge of the pedestrian is sampled. However, the lateral velocity calculation unit 304 may arbitrarily change the sampled edge to the right or left side depending on the direction in which the three-dimensional object (pedestrian) to be identified is expected to jump into the vehicle's lane.

[0033] Next, the control processing unit 108 performs a vehicle control process (S5) taking into consideration the result of identifying the three-dimensional object and the state of the vehicle (speed, steering angle, etc.). Through this vehicle control process (S5), the control processing unit 108 issues a warning to the vehicle occupants and performs braking of the vehicle, such as braking and adjusting the steering angle. The control processing unit 108 also determines a policy for performing object avoidance control by braking, and outputs the result to the in-vehicle network CAN 110 via the CAN IF 107.

[0034] The three-dimensional object identification process (S4) shown in Fig. 2 is performed by the arithmetic processing unit 105 in Fig. 1, and the vehicle control process (S5) is performed by the control processing unit 108 in Fig. 1. Output to the in-vehicle network CAN 110 is performed by the CAN IF 107. Each of these processes and each functional unit is configured, for example, by a single or multiple computer units, and is configured to be able to exchange data with each other.

[0035] In addition, the disparity or distance between each pixel of the left and right images is obtained by the parallax processing (S2) performed by the image processing unit 104. Therefore, in the three-dimensional object detection processing (S3) performed by the three-dimensional object detection unit 301, pixels with the same disparity or distance are grouped as three-dimensional objects in three-dimensional space. Note that the three-dimensional object detection processing (S3) does not detect the type of three-dimensional object, so if a person is standing near a tree, the tree and the person may be grouped together. Furthermore, if multiple people are standing at the same distance, the multiple people may be grouped together. After the three-dimensional object detection processing (S3), a three-dimensional object identification processing (S4) is performed based on the position on the image and the detection area for detecting the three-dimensional object.

[0036] 3 is a flowchart showing an example of a process for calculating the lateral velocity of a three-dimensional object, which is the basis of the present invention. Here, the process will be explained focusing on the three-dimensional object detection process (S3), various identification processes (S4), and vehicle control process (S5) shown in FIG.

[0037] First, the three-dimensional object detection unit 301 acquires an image obtained by the stereo camera or monocular camera from the image data buffer 203 (S31). Next, the three-dimensional object detection unit 301 detects a three-dimensional object appearing in the image (S32). Next, the three-dimensional object size calculation unit 302 calculates the size of the detected three-dimensional object (S33).

[0038] Next, the three-dimensional object identification unit 303 performs an identification process for the detected three-dimensional object (S41) and determines whether the three-dimensional object is a pedestrian, which is the object to be identified. If the three-dimensional object is a pedestrian, the lateral speed calculation unit 304 calculates the lateral speed of the pedestrian (S42). Next, the control processing unit 108 performs a vehicle control process (S5), and ends this process.

[0039] The lateral speed calculated in step S42 is the speed in the x direction when the longitudinal axis of the host vehicle is the z axis and the lateral axis of the host vehicle is the x axis, and is the lateral component of the pedestrian's relative speed with respect to the host vehicle. If a pedestrian is present in the host vehicle's lane, or if it is determined that a pedestrian not in the host vehicle's lane is moving at the lateral speed calculated above and running into the host vehicle's lane, a warning or brake control is issued in the vehicle control process (S16), thereby supporting the safety of the driver and pedestrian.

[0040] <Explanation of Process for Calculating Lateral Speed ​​of Pedestrian> Next, an example of process for calculating the lateral speed of a pedestrian using images obtained from a stereo camera will be described.

[0041] Fig. 4 is a diagram showing the results of a three-dimensional object detected based on a camera image by the three-dimensional object detection process (S3) shown in Fig. 2. Here, the details of the three-dimensional object detection process will be described using an image 401 acquired from a stereo camera by the three-dimensional object detection unit 301 shown in Fig. 2 as an example.

[0042] When the three-dimensional object detection unit 301 detects a three-dimensional object in the image 401, a plurality of windows 402 and 403 of different sizes are set in the image 401. The windows 402 and 403 are areas in which the three-dimensional object identification unit 303 identifies whether or not the three-dimensional object detected at each position in the image 401 by the three-dimensional object detection unit 301 is a pedestrian.

[0043] As described above, the distance to a three-dimensional object is determined by the parallax processing (S2) shown in Fig. 2. Therefore, the three-dimensional object detection unit 301 can accurately identify three-dimensional objects at different distances by using a large window 402 for three-dimensional objects close to the vehicle and a small window 403 for three-dimensional objects far from the vehicle.

[0044] The three-dimensional object detection unit 301 slides windows 402 and 403 within the image 401, and performs a three-dimensional object identification process (scanning process) at each position to which the windows are slid a predetermined distance. When a three-dimensional object is detected, rectangular detection areas 404 and 405 are displayed at the detected locations of the three-dimensional object within the image 401.

[0045] The detection areas 404 and 405, which are the detection results of the three-dimensional object detection process (S3), indicate areas where three-dimensional objects such as pedestrians and vehicles existing in three-dimensional space are projected as areas on the image. The three-dimensional object identification unit 303 reduces processing costs by identifying whether the three-dimensional object is a pedestrian or not based on the detection areas 404 and 405.

[0046] The detection areas 404 and 405 may be rectangular or may be an indeterminate shape that changes depending on the parallax and distance. However, they are generally treated as rectangular to facilitate computer handling in the three-dimensional object identification process (S4). For this reason, the following details of each process will be explained assuming that the three-dimensional object identification unit 303 identifies a three-dimensional object using the rectangular detection area 404.

[0047] Next, a conventional lateral speed calculation process and the calculation results will be described. Figures 5A to 5C are diagrams showing examples of the conventional speed calculation process and the calculation results. Pedestrian identification processing is performed based on a detection area 404 calculated from an image 401 obtained from a stereo camera.

[0048] 5A is a diagram showing an example of a right edge 501 of a pedestrian 500 captured in the detection area 404. When the identification result indicates a pedestrian 500, the process proceeds to the lateral speed calculation process (S15) shown in FIG. 3. A conventional calculation processing unit refers to the left and right edges of the pedestrian 500 to calculate the lateral speed of the pedestrian 500. In the following explanation, an example in which the right edge 501 of the pedestrian 500 is referenced is shown. In the figure, the right edge 501 is represented by a thick solid curve as the outline of the right side of the pedestrian 500.

[0049] 5B is a diagram showing an example of the average horizontal position 502 of the right edge 501 calculated from the t-th frame image 401. A conventional calculation processing unit, for example, acquires the horizontal position of each pixel of the right edge 501 and calculates the average horizontal position value. The average horizontal position value is represented by a thick solid line as the average horizontal position 502. For example, the average horizontal position value is calculated by calculating the average horizontal position in the detection region 404 based on the horizontal position coordinates of multiple pixels sampled from the right edge 501.

[0050] 5C is a diagram showing an example of the average horizontal position 502 of the right edge 501 calculated from the (t+1)th frame image 401. In the (t+1)th frame, the average horizontal position 502 of the right edge 501 is calculated using the same process as in the tth frame shown in FIG. 5B. Here, if the pedestrian 500 is moving to the right, a movement amount 504 from the average horizontal position 502 to the average horizontal position 503 is calculated from the tth frame to the (t+1)th frame.

[0051] In this way, the conventional calculation processing unit calculates the average lateral position 502 of the previous frame and the average lateral position 502 of the current frame. Then, the conventional calculation processing unit calculates the amount of movement 504 of the average lateral position 502 for each frame, and calculates the lateral speed of the pedestrian 500 based on the time between frames and the amount of movement 504. It is clear that the larger the amount of movement 504, the faster the pedestrian 500 is moving in the direction of movement of the average lateral position 502.

[0052] Next, a conventional problem that arises when calculating the horizontal position average will be described. Figures 6A to 6C are diagrams showing examples of images captured when a pedestrian 600 is standing near a tall, three-dimensional object such as a utility pole.

[0053] FIG. 6A is a diagram showing an example of a detection result when a non-target object 601 is present near a pedestrian 600. The image captured in FIG. 6A is considered to be the first frame. In the three-dimensional object detection process (S3) shown in FIG. 3, when a non-target object 601 other than the pedestrian to be identified, such as a utility pole or a signpost, is present near the pedestrian 600, an event occurs in which a detection area 602 larger than the pedestrian 600 is erroneously detected. In this case, the detection area 602 includes an edge position 603 of the non-target object 601, resulting in noise being added to the edge position of the pedestrian 600. As a result, the horizontal position average 604 shown by the solid line is calculated at a position different from the ideal horizontal position average 605 shown by the dashed line.

[0054] FIG. 6B is a diagram showing an example of the detection result for the second frame. The image captured in FIG. 6B is the second frame. In both the first and second frames, the pedestrian 600 actually remains stationary. However, suppose that the vehicle moves forward, and the non-target object 601 no longer exists within the detection area in the second frame. In this case, only the edge 606 for the pedestrian 600 is calculated, and the edge 606 does not include the edge of the non-target object 601. For this reason, the conventional arithmetic processing unit calculates the average lateral position 607 for the edge 606.

[0055] 6C is a diagram showing an example of the amount of movement of the average lateral position. The amount of movement 608 of the average lateral position 607 calculated in the second frame is calculated to be larger than the average lateral position 604 calculated in the first frame. This amount of movement 608 causes the lateral speed of the pedestrian 600 to be erroneously calculated as being larger, even though the pedestrian 600 is stopped. For example, a situation like this may occur when the pedestrian and a utility pole appear to overlap at one timing while the host vehicle is turning (e.g., turning right or left), but the pedestrian and the utility pole overlap and disappear at the next timing. Another situation may occur when the pedestrian and the utility pole appear to overlap when the host vehicle is far away from the pedestrian, but as the host vehicle approaches the pedestrian, the pedestrian and the utility pole overlap and disappear.

[0056] As described above, the conventional movement amount calculation process has a problem in that a pedestrian 600 who is outside the range of the own lane and is actually stopped is mistakenly determined to have jumped into the own lane, which can lead to a false alarm or false braking. For this reason, it is necessary to correctly calculate the movement amount of the pedestrian even when a non-target object 601 is present within the detection area.

[0057] Next, a movement amount calculation process according to the first embodiment will be described. Figures 7A and 7B are diagrams showing an example of the movement amount calculation process according to the first embodiment. Figures 7A and 7B also illustrate a case where a pedestrian 600 is standing near a tall object such as a utility pole.

[0058] The three-dimensional object detection unit 301 of the in-vehicle stereo camera device 100 according to the first embodiment first calculates the physical quantity of the size of a three-dimensional object from the size of the detection area on the image and the distance. Because the size of the three-dimensional object (especially a pedestrian) detected by the three-dimensional object detection unit 301 is unstable, even when a detection area 602 larger than the pedestrian 600 is detected, as shown in Fig. 6A, the lateral speed calculation unit 304 needs to suppress erroneous calculation of the speed of the pedestrian 600. Here, two methods will be described in which the lateral speed calculation unit 304 uses the size of the three-dimensional object to set the sampling range of the edge used for the lateral speed.

[0059] (First Method for Setting Edge Sampling Range) First, an edge sampling method (first method) using a pre-estimated size of a pedestrian will be described with reference to FIGS. 7A and 7B.

[0060] Fig. 7A is an example of a detection area 602 including a pedestrian 600. Fig. 7A shows an example in which the first frame image is acquired.

[0061] Because the pedestrian 600 was near a non-target object 601 (such as a utility pole), it is assumed that the three-dimensional object detection unit 301 detected a detection area 602 that was larger than the pedestrian 600. Therefore, the lateral speed calculation unit 304 references the physical quantity of the vertical width of the detection area 602, samples edges within an area 701 that excludes portions of the road surface that are equal to or greater than a predetermined value, and calculates an average lateral position 702. The predetermined value of the height from the road surface is set to, for example, 2.0 m. Alternatively, the predetermined value of the height from the road surface may be set to, for example, 2.1 m or 2.2 m. The predetermined value of the height from the road surface is set based on the height of a human being. The movement information calculation unit (lateral speed calculation unit 304) sets a sampling range that excludes areas in the image whose height from the road surface exceeds a predetermined value.

[0062] Fig. 7B is an enlarged view of the right side of the pedestrian 600 captured in the second frame. Fig. 7B shows an example in which the image in the second frame is captured.

[0063] No non-target object 601 is present within the detection area 602 of the image of the second frame. Therefore, as described with reference to FIG. 6B , the lateral velocity calculation unit 304 calculates the average lateral position 607 using only the right edge of the pedestrian 600. As a result, the amount of movement 703 from the average lateral position 702 calculated in the first frame to the average lateral position 607 calculated in the second frame is much smaller than the amount of movement 608 calculated in FIG. 6C . In this way, the three-dimensional object detected from the images of multiple frames is identified as the pedestrian 600, allowing the lateral velocity calculation unit 304 to accurately calculate the lateral velocity of the pedestrian 600.

[0064] (Second Method for Setting Edge Sampling Range) Next, a method (second method) for edge sampling using the identification result will be described with reference to FIGS. 8A and 8B. FIG.

[0065] Fig. 8A is a diagram showing an example of a detection region 710 including a pedestrian 600. Fig. 8A shows an example in which the first frame image is acquired.

[0066] The three-dimensional object identification unit 303 associates the identification result of the three-dimensional object identified in the first frame with the three-dimensional object size calculated by the three-dimensional object size calculation unit 302 and stores it in the storage unit 106 so that the three-dimensional object identification unit 303 can refer to it when processing the next frame. Next, the three-dimensional object identification unit 303 calculates an identification score for the detection area 710 in the three-dimensional object identification process (S41) of Fig. 3. For example, if the minimum identification score is 0 points and the maximum identification score is 100 points, as shown in Fig. 8A, if the detection area 710 has a high proportion of pedestrians 600, who are the identification target, then a high identification score will be assigned to this detection area 710.

[0067] If the detection process of the identification target by the three-dimensional object detection unit 301 is stable, the identification accuracy of the identification target by the three-dimensional object identification unit 303 will be high. For example, if the identification score calculated frame by frame for the detection area 710 is high, the detection process of the identification target is stable. Therefore, it can be assumed that the three-dimensional object size calculated by the three-dimensional object size calculation unit 302 based on the detection area 710 is also stable. In the image of the first frame shown in FIG. 8A , it is assumed that the identification score of the pedestrian 600 is high due to the stable detection area 710.

[0068] 8B is a diagram showing an example in which a detection area 602 larger than the identification target is acquired. In FIG. 8B, an example in which a second frame image is acquired is shown.

[0069] For example, suppose that after the three-dimensional object detection unit 301 acquires the first frame of an image, the setting of the detection area becomes unstable, causing the three-dimensional object detection unit 301 to acquire a detection area 602 in the next frame that is larger than the identification target (pedestrian 600). In this case, the edge position 603 of the non-target object 601 is included above the detection area 602, and the blank area other than the pedestrian 600 is larger than the detection area 710 shown in Figure 8A. For this reason, the lateral speed calculation unit 304 cannot calculate the average lateral position that is desirable for the edge of the pedestrian 600.

[0070] Therefore, the movement information calculation unit (lateral velocity calculation unit 304) sets a sampling range based on the identification result indicating that a three-dimensional object is an identification target and the size of the three-dimensional object calculated in the image of the frame in which the identification result was obtained. For example, the three-dimensional object size 704 in the detection area 710 of the first frame in which the lateral velocity calculation unit 304 determined that the identification score was high in the image of FIG. 8A is set as the sampling range of the edge used for lateral velocity calculation in the next frame, the second frame. By calculating the average lateral position 705 within the sampling range set based on the three-dimensional object size 704, the lateral velocity calculation unit 304 can prevent the movement amount of the pedestrian 600 from being calculated as being larger than the actual movement amount.

[0071] The second method uses an identification score, so even if, for example, the detection process becomes unstable and a large detection area 602 is obtained in the second frame image, the edge of the three-dimensional object can be sampled using a sampling range set based on the detection area 710 with a high identification score in the first frame image.

[0072] Furthermore, the second method does not set a predetermined distance from the road surface, as described in the first method. Therefore, the height of the detection area 710 with a high identification score is variable, and may be 1.5 m or 1.8 m. However, since there is no need to set a predetermined distance from the road surface in advance, this method is more versatile than the first method.

[0073] By using the first or second method described above, the lateral speed calculation unit 304 can suppress an increase in the average lateral position movement amount 703 and prevent an erroneous speed calculation for a stopped pedestrian 600.

[0074] FIG. 9 is a flowchart showing an example of detailed processing of the three-dimensional object identification processing (S4) in FIG.

[0075] First, the three-dimensional object identification unit 303 performs a pedestrian identification process (S51) for the detection area obtained in the three-dimensional object detection process (S3) in the three-dimensional object identification process (S41) shown in Fig. 3. In the pedestrian identification process, the three-dimensional object identification unit 303 determines whether the three-dimensional object detected in the detection area is a pedestrian based on the identification results of multiple frames (S52). If the three-dimensional object identification unit 303 determines that the object to be identified is not a pedestrian (NO in S52), it ends this process.

[0076] On the other hand, if the three-dimensional object identification unit 303 determines that the object to be identified is a pedestrian (YES in S52), the process proceeds to a lateral velocity calculation process (S42). In the lateral velocity calculation process (S42), the lateral velocity calculation unit 304 calculates the left and right edges of the pedestrian (S61).

[0077] Next, the lateral velocity calculation unit 304 calculates the size of the three-dimensional object, that is, the pedestrian (S62). In the three-dimensional object size calculation process, the lateral velocity calculation unit 304 calculates a physical quantity (e.g., height) of the three-dimensional object size based on the detection area on the image and the distance. Note that the three-dimensional object size calculated in step S62 is the size of a pedestrian, but the three-dimensional object size calculated by the three-dimensional object size calculation unit 302 shown in FIG. 2 is the size of a three-dimensional object that is not known to be a pedestrian. Also, instead of calculating the three-dimensional object size in step S62, the lateral velocity calculation unit 304 may obtain the three-dimensional object size calculated in step S33 of FIG. 3 from the three-dimensional object size calculation unit 302.

[0078] Next, the lateral velocity calculation unit 304 samples the left and right edges of the three-dimensional object using the three-dimensional object size calculated in step S62 (S63), and then calculates the average lateral position of the sampled left and right edges (S64).

[0079] Next, the lateral velocity calculation unit 304 calculates the amount of movement of the average lateral positions of the left and right edges in the current frame based on the average lateral positions of the left and right edges calculated in the past frame and the average lateral positions of the left and right edges in the current frame (S65). Finally, the lateral velocity calculation unit 304 calculates the lateral velocity of the pedestrian (S66), and this process ends.

[0080] <Example of Hardware Configuration of Computer> Next, the hardware configuration of the computer 800 that constitutes each device of the in-vehicle stereo camera device 100 will be described.

[0081] 10 is a block diagram showing an example of the hardware configuration of the calculator 800. The calculator 800 is an example of hardware used as a computer that can operate as the in-vehicle stereo camera device 100 according to this embodiment. The in-vehicle stereo camera device 100 according to this embodiment realizes various processes that are performed by the respective functional blocks shown in FIGS. 1 and 2 in cooperation with each other as the calculator 800 (computer) executes a program.

[0082] The computer 800 includes a CPU (Central Processing Unit) 801, a ROM (Read Only Memory) 802, and a RAM (Random Access Memory) 803, each connected to a bus 804. The computer 800 further includes a non-volatile storage 805 and a network interface 806.

[0083] The CPU 801 reads out program code of software that realizes each function according to this embodiment from the ROM 802, loads it into the RAM 803, and executes it. Variables, parameters, etc. generated during the calculation processing of the CPU 801 are temporarily written to the RAM 803, and these variables, parameters, etc. are read out as appropriate by the CPU 801. However, instead of the CPU 801, an MPU (Micro Processing Unit) or a GPU (Graphics Processing Unit) may be used, or the CPU 801 and a GPU (Graphics Processing Unit) may be used together.

[0084] The nonvolatile storage 805 may be, for example, a hard disk drive (HDD), a solid state drive (SSD), a flexible disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, or a nonvolatile memory. In addition to an operating system (OS) and various parameters, programs for operating the computer 800 are recorded in this nonvolatile storage 805. The ROM 802 and the nonvolatile storage 805 record programs, data, and the like necessary for the CPU 801 to operate, and are used as an example of a computer-readable, non-transitory storage medium that stores programs executed by the computer 800.

[0085] The network interface 806 may be, for example, a NIC (Network Interface Card), and various data may be transmitted and received between devices via a LAN (Local Area Network), dedicated line, etc. connected to the NIC terminal.

[0086] The in-vehicle stereo camera device 100 according to the first embodiment described above calculates parallax information from images captured by the stereo camera when a stereo camera is used as a sensor for detecting external information, and acquires an image of a three-dimensional object. If the in-vehicle stereo camera device 100 subsequently identifies a three-dimensional object in the image as a pedestrian, it calculates the left and right edges of the pedestrian and calculates the size of the three-dimensional object based on the left and right edges. If the three-dimensional object is identified as a pedestrian, the sampling range is set based on the three-dimensional object size, so that edges of non-target objects close to the pedestrian in the image are not sampled. As a result, highly accurate lateral speed calculations are possible even when detection of the three-dimensional object being identified is unstable. Furthermore, improved accuracy in calculating the lateral speed improves the accuracy of warnings or brake control to prevent the vehicle from colliding with the three-dimensional object, leading to safety assistance for the driver and pedestrians.

[0087] Furthermore, whether or not a three-dimensional object is a target for identification is identified based on images captured in multiple frames. Therefore, even if a three-dimensional object appears close to a non-target object in an image of a previous frame, the three-dimensional object identification unit 303 can accurately identify the three-dimensional object as a target for identification if the three-dimensional object appears far away from the non-target object in an image of a subsequent frame.

[0088] In the first method, the height from the road surface in the image is set to the average height of the pedestrian to be identified, and the sampling range is set to exclude areas that exceed the height from the road surface. As a result, the lateral speed calculation unit 304 can prevent non-target objects that are taller than the target object from being sampled, thereby preventing the lateral position value calculated based on the non-target object that is taller than the target object from being included in the average lateral position value.

[0089] In the second method, the identification result indicating that a three-dimensional object is an identification target is expressed as an identification score. The higher the identification score, the more stable the detection process for the identification target is. Therefore, the three-dimensional object size calculated based on the detection area with a high identification score is also stable, and the sampling range is set based on that three-dimensional object size.

[0090] The in-vehicle stereo camera device 100 according to this embodiment can accurately calculate the movement of a pedestrian even when the pedestrian is walking somewhere other than a crosswalk.

[0091] In addition, in the above-described embodiment, the object to be identified is a pedestrian, but the object to be identified may also be at least one of a person riding a bicycle, a person riding a motorbike, or a person riding a personal mobility vehicle.

[0092] Furthermore, in the above-described embodiment, the lateral velocity calculation unit 304 uniformly samples either the left or right edge of the pedestrian 600, but the sampled information may be weighted. For example, when the pedestrian 600 is moving, the lower body moves more than the upper body, and therefore, the movement of the lower body is likely to cause an error in the average lateral position. Therefore, the lateral velocity calculation unit 304 weights the sampled information of the upper body of the pedestrian 600 more heavily than the lower body, thereby enabling the average lateral position of the pedestrian 600 to be accurately determined.

[0093] Second Embodiment Next, an example of the configuration of an in-vehicle camera device according to a second embodiment of the present invention will be described with reference to Fig. 11. The in-vehicle camera device according to the second embodiment is configured to use a monocular camera and millimeter-wave radar as sensors, instead of the configuration in which a stereo camera is used as a sensor in the first embodiment.

[0094] 11 is a block diagram showing an example configuration of an in-vehicle camera device 900. The in-vehicle camera device 900 includes a monocular camera 901, a millimeter-wave radar 902, an image data buffer 903, a ranging data buffer 904, and a CAN IF 910. The in-vehicle camera device 900 has the same functional blocks as the in-vehicle stereo camera device 100 shown in FIG. 1. Therefore, the monocular camera 901 is connected to an image input IF 103 (see FIG. 1), and image data input through the image input IF 103 is stored in the storage unit 106 via the internal bus 109. An input IF (not shown) for the millimeter-wave radar 902 is also provided, and data output by the millimeter-wave radar 902 is input through this input IF and stored in the storage unit 106 via the internal bus 109.

[0095] The monocular camera (monocular camera 901) is an example of an imaging unit having one imaging element, and captures an image of the environment in front of the vehicle. Image data of the image captured by the monocular camera 901 is subjected to image processing (S21) by the image processing unit 104. The content of the image processing (S21) is similar to the image processing (S1) shown in FIG. 2. The image data after the image processing is buffered in an image data buffer 903 provided in the storage unit 106. Note that, unlike the first embodiment described above, the imaging unit is not configured as a stereo camera, and therefore the in-vehicle camera device 900 according to the second embodiment does not perform the parallax processing (S2) shown in FIG. 2.

[0096] The distance measurement unit (millimeter-wave radar 902) measures the distance to a three-dimensional object. For example, the millimeter-wave radar 902 irradiates millimeter-wave band radio waves ahead of the vehicle and measures the distance to an object that reflects the radio waves, the speed of the object, etc. The data output by the millimeter-wave radar 902 is subjected to distance measurement processing (S22) by the arithmetic processing unit 105. The processed distance measurement data is buffered in a distance measurement data buffer 904 provided in the storage unit 106.

[0097] Thereafter, a three-dimensional object detection process (S23) is performed using the image data read from the image data buffer 903 and the distance measurement data read from the distance measurement data buffer 904. The three-dimensional object detection process (S23) corresponds to the three-dimensional object detection process (S3) shown in Fig. 2, and the detection areas 404 and 405 shown in Fig. 4 are switched and used according to the distance measurement data to detect a three-dimensional object.

[0098] The three-dimensional object detection process (S23), followed by the three-dimensional object identification process (S24) and vehicle control process (S25) correspond to the three-dimensional object identification process (S4) and vehicle control process (S5) shown in Figures 2 and 3. For example, the three-dimensional object detection unit (three-dimensional object detection unit 301) detects a three-dimensional object at a specific distance based on an image input from an imaging element. The three-dimensional object identification unit (three-dimensional object identification unit 303) identifies the detected three-dimensional object. In the three-dimensional object size calculation process (S33) of the three-dimensional object detection process (S3) shown in Figure 3, the three-dimensional object size calculation unit 302 can calculate the three-dimensional object size based on the image obtained from the monocular camera 901 and the distance and detection size obtained by the millimeter-wave radar 902.

[0099] The CAN IF 910 outputs information output from the vehicle-mounted camera device 900 to a control system (not shown) of the vehicle itself via the vehicle-mounted network CAN 110 shown in FIG.

[0100] Furthermore, in addition to the monocular camera 901 and millimeter wave radar 902, the present invention can also be applied to devices equipped with sensors that can simultaneously measure images and distances.

[0101] In the in-vehicle camera device 900 according to the second embodiment described above, similarly to the in-vehicle stereo camera device 100 according to the first embodiment, when a three-dimensional object detected in an image is identified as a pedestrian, the lateral speed can be calculated using the first and second methods described above. Therefore, even if there is a non-target object such as a tall utility pole near the pedestrian, the lateral speed of the pedestrian can be accurately calculated.

[0102] The present invention is not limited to the above-described embodiments, and various other applications and modifications are possible without departing from the spirit and scope of the present invention as defined in the claims. For example, the above-described embodiments provide detailed and specific descriptions of the configuration of the environment recognition device in order to clearly explain the present invention, and the device is not necessarily limited to having all of the described components. Furthermore, it is possible to replace some of the components of the embodiments described here with components of other embodiments, and it is also possible to add components of other embodiments to the configuration of one embodiment. Furthermore, it is also possible to add, delete, or replace other components with respect to some of the configuration of each embodiment. Furthermore, the control lines and information lines shown are those considered necessary for the explanation, and not all control lines and information lines are necessarily shown in the actual product. In reality, it is reasonable to assume that almost all components are interconnected.

[0103] 100...In-vehicle stereo camera device, 101...Left camera, 102...Right camera, 103...Image input IF, 104...Image processing unit, 105...Calculation processing unit, 106...Memory unit, 108...Control processing unit, 109...Internal bus, 110...In-vehicle network CAN, 203...Image data buffer, 301...Three-dimensional object detection unit, 302...Three-dimensional object size calculation unit, 303...Three-dimensional object identification unit, 304...Lateral speed calculation unit

Claims

1. An environmental recognition device comprising: a three-dimensional object detection unit that sets a detection area for detecting an identification target in an image captured by an imaging unit, and detects three-dimensional objects appearing in the image based on the detection area; a three-dimensional object size calculation unit that calculates the size of the three-dimensional object detected in the detection area using information on the distance from the device to the three-dimensional object calculated from the image; a three-dimensional object identification unit that identifies the three-dimensional object detected in the detection area as the identification target; and a movement information calculation unit that limits a sampling range of the three-dimensional object based on the size of the three-dimensional object, and calculates movement information of the three-dimensional object that is the identification target moving laterally based on information on the edges of the three-dimensional object that is the identification target.

2. The environmental recognition device described in claim 1, wherein the three-dimensional object identification unit identifies the three-dimensional object as the identification target based on the images of multiple consecutive frames, and the movement information calculation unit calculates, for each frame, the average horizontal coordinate of the points where the edge of the three-dimensional object is sampled as the edge information, and calculates, as the movement information, the lateral speed of the three-dimensional object that is the identification target.

3. The environment recognition device according to claim 2, wherein the movement information calculation unit sets the sampling range excluding an area in the image whose height from the road surface exceeds a predetermined value.

4. The environmental recognition device according to claim 2, wherein the movement information calculation unit sets the sampling range based on an identification result indicating that the three-dimensional object is the identification target and the size of the three-dimensional object calculated in the image of the frame in which the identification result was obtained.

5. The environment recognition device according to claim 3, wherein the movement information calculation unit samples either the left or right edge of the three-dimensional object, and calculates an average value of the horizontal coordinates of the points where the edge is sampled.

6. The environmental recognition device described in claim 5, wherein the imaging unit is a stereo camera having two imaging elements, and includes an image processing unit that calculates the parallax of the three-dimensional object based on images input from each of the imaging elements, the three-dimensional object detection unit detects the three-dimensional object when the distance from the device itself and the vehicle on which the stereo camera is mounted is the same, and the three-dimensional object identification unit identifies the three-dimensional object when the distance is the same.

7. The environmental recognition device described in claim 5, wherein the imaging unit is a monocular camera having one imaging element, and is equipped with a distance measuring unit that measures the distance to the three-dimensional object, the three-dimensional object detection unit detects the three-dimensional object at a specific distance based on an image input from the imaging element, and the three-dimensional object identification unit identifies the detected three-dimensional object.

8. The environment recognition device according to claim 5, wherein the three-dimensional object to be identified is at least one of a pedestrian, a person riding a bicycle, a person riding a motorcycle, and a person riding a personal mobility vehicle.

9. The environment recognition device according to claim 5, wherein the lateral direction is a direction in which a pedestrian moves across the front of the vehicle on which the device is mounted, or the lane in which the vehicle is traveling.

10. An environmental recognition method comprising the steps of: setting a detection area for detecting an identification target in an image captured by an imaging unit, and detecting a three-dimensional object appearing in the image based on the detection area; calculating the size of the three-dimensional object detected in the detection area using information on the distance from the device to the three-dimensional object calculated from the image; identifying that the three-dimensional object detected in the detection area is the identification target; and limiting a sampling range of the three-dimensional object based on the size of the three-dimensional object, and calculating movement information of the three-dimensional object moving laterally based on information on the edges of the three-dimensional object that is the identification target.

Citation Information

Patent Citations

  • Jumping-out pedestrian determining device and program

    JP2012203884A

  • On-vehicle environment recognizing device

    JP2019106026A

  • Image processing device and image processing method

    WO2022113470A1