Outdoor environment detection device and outdoor environment detection method

Abstract

Outdoor environment detection device (130), comprising: an image acquisition unit for obtaining an image in a detection area (122); a grouping unit (162) for generating a section group by grouping, based on a first relative ratio between sections, a plurality of sections in an area extending from a plane corresponding to a road surface to a predetermined height in the obtained image; and a person candidate determination unit (164) for dividing the section group into two in a horizontal direction of the image and determining, based on a second relative ratio between two subdivided section groups, whether the section group is a first person candidate that is a candidate of a person (208), wherein the second relative relationship is whether or not each midline angle formed from a vertical line (218) and a midline (216a, 216b) of each of the two subdivided section groups is contained in a predetermined first angle range.

Description

[0001] The invention relates to an external environment recognition device and an external environment recognition method for recognizing the environment outside a subject vehicle.

[0002] A technology that specifies a target object such as a vehicle, a person, and an obstacle located in front of a subject vehicle to avoid a collision with the specified target object (brake control) and performs control to maintain a safe distance between the subject vehicle and the preceding vehicle (cruise control) is known (for example, Japanese Patent No. 3349060). In many of these technologies, a target object is tracked to update position information, and the moving speed and the like of the preceding vehicle and the person are calculated. The result of this calculation is reflected in, for example, brake control and cruise control.

[0003] Among these target objects, a person, especially a pedestrian, is highly flexible in terms of movement direction, their movement speed is low compared to a vehicle ahead, and their absolute volume is small. Therefore, it is difficult to detect and track the person's behavior, making it difficult to apply braking control and cruise control technologies.

[0004] Accordingly, a technology has been proposed in which, in an image acquired by the subject vehicle, an area where motion is uniform is provisionally defined as an upper area, and a determination is made as to whether the motions of feature points in a lower portion are non-uniform. If the motions are determined to be non-uniform, the area is defined as a lower area, and a combination thereof is specified as an entire body area of ​​a pedestrian (for example, Japanese Unexamined Patent Application (JP 2009-157581 A).

[0005] When the technology of Japanese Patent JP 3 349 060 B2 is used, the entire body of a person can be specified during daylight. However, in a condition where it is necessary to turn on the headlights, such as at night, an effective image can only be obtained at a relatively low position of the entire body area of ​​the pedestrian (the lower body area) on which the headlights of the subject vehicle are incident, and it is impossible to specify a region where the movements are uniform, which corresponds to the upper body area. Therefore, for example, the entire body area of ​​the pedestrian crossing the road in front of a subject vehicle cannot be specified, and this may cause problems with smooth braking control and cruise control.

[0006] In view of such problems, the object of the invention is to provide an external environment recognition device and an external environment recognition method capable of improving the efficiency of specifying a person and the accuracy of specifying a person by directly specifying the person with a lower portion area (the lower half of the body) of the person.

[0007] Examples and embodiments of the prior art are described in JP 2010 205087 A, US 7 702 019 B2, JP 2009 157581 A, US 2009 / 0041302 A1, US 7 233 233 B2 and WO 2008 / 065729 A1.

[0008] To achieve this object, according to one aspect of the present invention, there is provided an external environment recognition device including: an image acquisition unit for acquiring an image in a detection area, a grouping unit for generating a section group by grouping, based on a first relative relationship between sections, a plurality of sections in an area extending from a plane corresponding to the road surface to a predetermined height in the acquired image, and a person candidate determination unit for dividing the section group into two in a horizontal direction of the image and determining, based on a second relative relationship between two divided section groups, whether the section group is a first person candidate which is a candidate of a person, wherein the second relative relationship is whether or not each center line angle,formed by a vertical line and a center line of each of the two divided section groups, is contained in a predetermined first angular range.,

[0009] Preferably, the first relative relationship is one of a relative positional relationship, a difference of luminance, or a continuity in an edge direction.

[0010] Preferably, the second relative relationship is one or more relationships selected from a group including a shape, a size of an area, symmetry, and luminance.

[0011] Preferably, the grouping unit further divides, into a plurality of regions in a height direction, the region extending from the plane corresponding to the road surface to a predetermined height in the image, and groups a plurality of sections in each of the divided regions, and the person candidate determining unit determines the first person candidate when the second relative relationship between the two divided section groups is satisfied and a difference angle between center lines of the grouped section groups adjacent to each other in the height direction is within a predetermined second angle range.

[0012] Preferably, the person candidate determining unit does not perform determination of the first person candidate with respect to the remaining areas when a difference angle formed by any of the areas in the height direction is determined not to be included in the predetermined second angle range.

[0013] Preferably, the person candidate determination unit changes the first angular range according to a distance between a subject vehicle and the section group.

[0014] Preferably, the external environment recognition device comprises: a person candidate tracking unit for tracking the section group determined as the first person candidate by the person candidate determining unit, a reversal measuring unit for measuring the number of reversals when the person candidate determining unit determines that the section group tracked by the person candidate tracking unit is the first person candidate and is not the first person candidate, and a person candidate specifying unit for specifying the first person candidate as a second person candidate when the number of reversals is determined to be included in a predetermined frequency range.

[0015] Preferably, the external environment recognition device comprises: a person candidate tracking unit for tracking the section group determined by the person candidate determining unit to be the first person candidate, a reversal measuring unit for measuring a reversal cycle when the person candidate determining unit determines that the section group tracked by the person candidate tracking unit is the first person candidate and is not the first person candidate, and a person candidate specifying unit for determining that the first person candidate is moving when a reversal cycle is determined to be included within a predetermined cycle range.

[0016] According to a second aspect, the present invention provides an external environment recognition method comprising the steps of: acquiring an image in a detection area, generating a section group by grouping based on a first relative relationship between sections, wherein a plurality of sections extend in a range from a plane corresponding to a road surface to a predetermined height in the acquired image, dividing the section group into two in a horizontal direction of the image, and determining, based on a second relative relationship between two divided section groups, whether the section group is a first person candidate, which is a candidate of a person, wherein the second relative relationship is whether or not each center line angle formed by a vertical line and a center line of each of the two divided section groups,is contained in a predetermined first angular range.,

[0017] The invention is explained below using exemplary embodiments with reference to the attached drawings. Fig. 1 is a block diagram illustrating a wiring relationship in an environmental detection system; Fig. 2A and Fig. 2B are exemplary illustrations for explaining a luminance image; Fig. 3 is a functional block diagram schematically illustrating functions of an external environment detection device; Fig. 4 is an explanatory diagram for explaining the conversion into three-dimensional position information performed by a position information acquiring unit; Fig. 5 is an explanatory diagram for explaining an area limitation by a grouping unit; Fig. 6 is an explanatory diagram for explaining the operation of a person candidate determination unit; Fig. 7 is an explanatory diagram for explaining the operation of a person candidate determination unit; Fig. Fig. 8 is an explanatory diagram for explaining the division by the grouping unit; Fig. 9 is an explanatory diagram for explaining the operation of a reverse measuring unit; Fig. 10 is a flowchart for explaining the overall flow of an external environment detection method; Fig. 11 is a flowchart for explaining a flow of a position information acquisition process; Fig. 12 is a flowchart for explaining a flow of a grouping / person candidate determination process; Fig. 13 is a flowchart for explaining a flow of a grouping / person candidate determination process; Fig. 14 is a flowchart for explaining a flow of a grouping / person candidate determination process; Fig. 15 is a flowchart for explaining a flow of a grouping / person candidate determination process; Fig. 16 is a flowchart for explaining a flow of a person candidate tracking process; Fig. 17 is a flowchart for explaining a flow of a reverse measurement process; Fig. 18 is a flowchart for explaining a flow of a person candidate specification process; and Fig. Figure 19 is an example diagram to explain continuity in an edge direction.

[0018] Hereinafter, a preferred embodiment of the invention will be described in detail with reference to the accompanying drawings. The sizes, materials, and other specific numerical values ​​shown in the embodiment are merely examples for facilitating understanding of the invention, and unless otherwise specified, they do not limit the invention. In the description and drawings, elements having substantially the same functions and configurations are designated by the same reference numerals, and repeated explanations thereof are omitted. Elements not directly related to the invention are omitted from the drawings. (Environmental detection system 100)

[0019] Fig. 1 is a block diagram illustrating a wiring relationship in an environment recognition system 100. The environment recognition system 100 includes a plurality of image pickup devices 110 (two image pickup devices 110 in the present embodiment), an image processing device 120, an external environment recognition device 130, and a vehicle control device 140 provided in a vehicle 1.

[0020] The image pickup devices 110 include an imaging element such as a CCD (charge-coupled device) or a CMOS (complementary metal oxide semiconductor) and can obtain a color image, that is, luminances of three color phases R (red), G (green), B (blue) per pixel. In this case, a color image captured by the image pickup devices 110 is called a luminance image and is different from a distance image, which will be explained later.

[0021] The image pickup devices 110 are arranged in a substantially horizontal direction at a distance from each other such that the optical axes of the two image pickup devices 110 are substantially parallel in a traveling direction of the vehicle 1. The image pickup device 110 continuously generates image data obtained by capturing an image of a specific object located in a detection range in front of the vehicle 1, for example, every 1 / 60 second (60 fps).

[0022] In this case, the specific object can be not only an independent three-dimensional one, such as a person (human), a vehicle, a traffic light, a road and a guardrail, but also a lighting part or section, such as a brake light (tail light), a turn signal or indicator light, a traffic light, which are things that can be specified as a part or section of a three-dimensional object.

[0023] The image processing device 120 receives image data from each of the two image pickup devices 110 and, based on the two pieces of image data, derives parallax information including a parallax of each block (a set of a predetermined number of pixels) in the image as a section and a position representing a position of each block in the image. The image processing device 120 derives parallax using so-called pattern matching, which searches for a block in one of the image data that corresponds to the block optionally extracted from the other image data. In this embodiment, the horizontal direction, when used to explain the block, means a horizontal direction for the captured image and corresponds to the width direction in the real world.On the other hand, “vertical” direction means a vertical direction for the captured image and corresponds to the height direction in the real world.

[0024] One way to perform pattern matching is to compare luminance values ​​(Y-color difference signals) between two pieces of image data of a block indicating an arbitrary image position. Examples include SAD (sum of absolute difference) that obtains a difference in luminance values, SSD (sum of squared intensity difference) that squares a difference, and NCC (normalized cross-correlation) that uses a similarity degree of distribution values ​​obtained by subtracting a mean luminance value from a luminance value of each pixel. The image processing device 120 performs such a parallax derivation process on all blocks that appear in the detection area (for example, 600 pixels x 200 pixels). In this case, the block includes 4 pixels x 4 pixels. However, the number of pixels in the block can be set to any other value.

[0025] Although the image processing device 120 can derive a parallax for each block serving as a detection resolution unit, it is impossible to know what type of target object the block belongs to. Therefore, the parallax information is not derived from the target object, but is independently derived by resolution (e.g., per block) in the detection range. In this embodiment, an image obtained by associating the thus-derived parallax information (corresponding to a relative distance described later) with image data is referred to as a distance image.

[0026] The Fig. 2A and Fig. 2B are exemplary illustrations for explaining a luminance image 124 and a distance image 126. For example, assume that the luminance image (image data) 124, as shown in Fig. 2A, with respect to a detection area 122 generated by the two image pickup devices 110. Here, for ease of understanding, only one of the two luminance images 124 is shown schematically. The image pickup device 120 obtains a parallax for each block from such a luminance image 124 and generates the distance image 126, as shown in Fig. 2B. Each block of the distance image 126 is assigned a parallax of that block. For explanatory purposes, a block from which a parallax is derived is indicated by a black dot in the drawing.

[0027] The parallax can be easily specified at the edge portion (a portion where there is a contrast between neighboring pixels) of the objects, and therefore the block from which the parallax is derived, which is marked with black dots in the distance image 126, is likely to also be an edge in the luminance image 124. Therefore, the Fig. 2A and the luminance image 124 shown in Fig. 2B with respect to the outline of each target object.

[0028] The external environment recognition device 130 obtains the luminance image 124 and the distance image 126 from the image processing device 120, and uses the luminance based on the luminance image 124 and the relative distance from the subject vehicle derived from the distance image 126 to determine which specific object the target object in the detection area 122 corresponds to. For example, by specifying a pedestrian (specific object) crossing the road in front of the subject vehicle, the existence and behavior of the pedestrian can be quickly detected, and this can be reflected in the braking control and cruise control.

[0029] Note that the above relative distance can be obtained by converting parallax information for each block in the distance image 126 into three-dimensional position information using a so-called stereo method. The stereo method is a method that uses a triangulation method to derive a relative distance of the target object with respect to the image pickup device 110 from the parallax of the target object. The process performed by the external environment recognition device 130 will be explained in detail later.

[0030] The vehicle control device 140 avoids collision with the target object specified by the external environment detection device 130 and performs control to maintain a safe distance from the preceding vehicle. Specifically, the vehicle control device 140 obtains the current traveling state of the vehicle 1 based on, for example, a steering angle sensor 142 for detecting a steering angle and a vehicle speed sensor 144 for detecting a speed of the vehicle 1, thereby driving an actuator 146 to maintain a safe distance from the preceding vehicle. The actuator 146 is an actuator for vehicle control used to control a brake, a throttle valve (torque control of the engine), a steering angle, or the like.When a collision with a target object is expected, the vehicle control device 140 displays a warning (notification) of the expected collision on a display 148 provided in front of a driver and controls the actuator 146 to automatically decelerate the vehicle 1. The vehicle control device 140 may also be implemented integrally with the external environment detection device 130. (External environment detection device 130)

[0031] Fig. 3 is a functional block diagram schematically illustrating functions of the external environment detection device 130. As shown in Fig. 3, the external environment detection device 130 includes an I / F unit 150, a data holding unit 152, a luminometer 154, and a central control unit 156.

[0032] The I / F unit 150 is an interface for interactive information exchange with the image processing device 120 and the vehicle control device 140. The I / F unit 150 functions as an image acquisition unit. The data holding unit 152 is formed, for example, by a RAM, a flash memory, an HDD, and the like, and holds or stores various types of information required for processing by the functional units explained below. Furthermore, the data holding unit 152 temporarily holds or stores the luminance image 124 and the distance image 126 received from the image processing device 120. The luminometer 154 measures the illumination intensity outside the vehicle to check the external environment.

[0033] The central control unit 156 includes a semiconductor integrated circuit, for example, including a central processing unit (CPU), a ROM that stores a program or the like, and a RAM that serves as a work area, and controls the I / F unit 150, the data holding unit 152, and the luminometer 154 through a system bus 158. In the present embodiment, the central control unit 156 also functions as a position information acquisition unit 160, a grouping unit 162, a person candidate determination unit 164, a person candidate tracking unit 166, a reverse measurement unit 168, and a person candidate specification unit 170. With each functional unit of the central control unit 156, as explained above, the external environment recognition device 130 specifies a target object in the detection area as a specific object.Depiction examples of specific objects include a preceding vehicle and a person (pedestrian). In this explanation regarding the present embodiment, for example, a person located in front of a vehicle 1 is specified in a situation where it is necessary to turn on the headlights, for example, at night (for the purpose of explanation, this will be referred to simply as "at night").

[0034] The position information acquisition unit 160 uses the stereo method to convert parallax information for each block in the detection area 122 of the distance image 126 into three-dimensional position information including a width direction x, a height direction y, and a depth direction z. The parallax information represents a parallax of each block in the distance image 126, while the three-dimensional position information represents information about the relative distance of each block in the real world. If the parallax information is derived not from the pixel but from the block, that is, the calculation can be performed in units of pixels with the parallax information assumed to be parallax information for all pixels belonging to a block.

[0035] Fig. 4 is an explanatory diagram for explaining the conversion into three-dimensional position information performed by the position information obtaining unit 160. First, the position information obtaining unit 160 treats the distance image 126 as a coordinate system in a pixel unit, as shown in Fig. 4 shown. In Fig. 4, the lower left corner is used as an origin (0,0). The horizontal direction is assumed to be an i-coordinate axis, and the vertical direction is assumed to be a j-coordinate axis. A pixel with a parallax dp can therefore be represented as (i, j, dp) using a pixel position i, j and the parallax dp.

[0036] The three-dimensional coordinate system in the real world according to the present embodiment is considered from a relative coordinate system in which the vehicle 1 is located at the center. The right side of the direction in which the vehicle 1 moves is referred to as a positive direction of the X-axis, the upper side of the vehicle 1 is referred to as a positive direction of the Y-axis, the direction in which the vehicle 1 moves (front side) is referred to as a positive direction of the Z-axis, and the intersection point between the road surface and a vertical line passing through the center of the two imaging devices 110 is referred to as an origin (0,0,0). When the road is assumed to be a flat surface, the road surface coincides with the XZ plane (y=0).The position information acquisition unit 160 uses Formulas 1 to 3 shown below to convert the coordinate of the block (i, j, dp) in the distance image 126 into a three-dimensional point (x, y, z) in the real world. x=CD / 2+z⋅PW⋅(i−IV) y=CH+z⋅PW⋅(j−JV) z=KS / dp

[0037] Here, CD denotes an interval (baseline length) between the image pickup devices 110, PW denotes a corresponding distance in the real world to a distance between adjacent pixels in the image, the so-called view angle per pixel, CH denotes an arrangement height of the image pickup device 110 from the road surface, IV and JV denote coordinates (pixels) in the image at an infinite point in front of the vehicle 1, and KS denotes a distance coefficient (KS=CD / PW).

[0038] Accordingly, the position information acquisition unit 160 derives the height from the road surface based on the detection distance (e.g., the number of pixels) in the distance image 126 between a point on the road surface located at the same relative distance as the block and the block. Note that an XZ plane corresponding to the road surface (y=0) can be derived based on a three-dimensional coordinate of a lane on the road (a line dividing the road), and the lane can be extracted from luminance changes with respect to the road. The three-dimensional position of the road surface can be derived using various types of available technologies, and a detailed description thereof is therefore omitted here.

[0039] The grouping unit 162 groups a plurality of blocks in the distance image 126 based on a relative positional relationship serving as the first relative relationship derived by the position information acquisition unit 160, and generates a block group serving as a section group. For example, the grouping unit 162 performs a grouping process such that a given block is assumed to be a base point, a block adjacent to this block is retrieved, and a block whose difference in a width direction x from the block serving as the base point, whose difference in a height direction y therefrom, and whose difference in a depth direction z therefrom are within a predetermined range (for example, 0.5 m) is assumed to correspond to the same target object.The range is represented as a real-world distance and can be set to any given value by the manufacturer and a driver.

[0040] The grouping unit 162 also groups blocks newly added through a grouping process, so that the grouping unit 162 further groups blocks whose difference in the width direction x from the block serving as the base point, whose difference in the height direction y from the block serving as the base point, and whose difference in the depth direction z from the block serving as the base point are within a predetermined range. As a result, all blocks provisionally determined to be the same specific object are grouped.

[0041] In the above description, the difference in the width direction x, the difference in the height direction y, and the difference in the depth direction z are determined independently, and only when these are all within the predetermined range are the target sections grouped into the same group. However, the grouping process can be performed using a different calculation. For example, if the Euclidean distance, the square root of ((difference in the width direction x) 2 + (Difference in the height direction y) 2 + (Difference in the depth direction z) 2 ) is within a predetermined range, target sections can be grouped into the same group. With such a calculation, distances between blocks in the real world can be accurately derived, thus increasing grouping accuracy.

[0042] In this case, three parameters, namely, the difference in the width direction x, the difference in the height direction y, and the difference in the depth direction z, are used. However, the embodiment is not limited to this. Alternatively, the grouping process may be performed using only two differences, such as the difference in the width direction x and the difference in the height direction y, the difference in the width direction x and the difference in the depth direction z, or the difference in the height direction y and the difference in the depth direction z. Furthermore, for a certain specific object, the overall luminance may be similar, and the tendency of the luminance change may be in a regular pattern, and therefore, the grouping unit 162 may make use of the luminance characteristic and additionally perform the grouping process using a condition that the luminance is close or similar.

[0043] In the present embodiment, when the external environment is determined to be at night using the luminometer 154, or when the headlights are in the ON state, the grouping unit 162 limits the detection target to a portion of the detection range 122. More specifically, the grouping unit 162 limits the range to an area extending from the plane corresponding to the road surface to a predetermined height (for example, 1.0 m), and groups only blocks located in this range.

[0044] Fig. Fig. 5 is an exemplary illustration for explaining the area limitation by the grouping unit 162. For the purpose of simple explanation, the luminance image 124 is used for explanation. The present embodiment is mainly intended to specify a person in front of a vehicle 1 at night. In this case, the vehicle 1 illuminates the road ahead using the headlights, but with a low beam used in a usual state, irradiating light onto the legs of a person, as in an area defined by a rectangle in Fig. 5A. Therefore, in a luminance image 124 taken at night, a target object of a height almost equal to that of a person's legs is mainly shown, and therefore, relative distances in the depth direction z are derived with respect to blocks that are lower than the height of the person's legs in the distance image 126. To handle such a night situation, the grouping unit 162 groups blocks only in a limited area 200, which is a three-dimensional belt shape of only one meter and which is located above the road surface, as shown by the diagonal lines in Fig. 5B shown.

[0045] As previously described, the detection target at night is limited to a range of one meter from the road surface, so this can prevent false detections, such as false detection of a target object located one meter or higher from the road surface, and reliably detect the person's legs. Furthermore, compared to processing where the entire detection area 122 is assumed as the target, the processing load can be significantly reduced.

[0046] In addition to the area extending from the plane corresponding to the road surface to the predetermined height (for example, 1.0 m), the grouping unit 162 may exclude an area outside a shoulder or guardrail, as shown in Fig. 5B. In this way, the load can be further reduced.

[0047] In this case, for example, nighttime or a nighttime condition is determined according to a measurement result provided by the luminometer 154 and the illumination state of the headlights, and the detection target is limited to the area within one meter of the road surface. However, when the external environment is bright, for example, during daylight, the detection target may be the entire detection area 122, so that the entire body of the person, including not only the lower half of the body but also the upper half of the body, can be subjected to the grouping process.

[0048] The person candidate determination unit 164 divides, in the horizontal direction of the image, some of the block groups grouped by the grouping unit 162 that can be determined to be a person's legs and whose area size is equal to or larger than a certain area size. Based on a second ratio between the block groups in two, the person candidate determination unit 164 determines whether or not the block groups are a first person candidate. Various types of ratios can be used as the second ratio. In this case, the total inclination of each of the two divided block groups is used as a typical example.Specifically, the person candidate determination unit 164 determines, as the second ratio, whether each centerline angle, which is an angle between a vertical line and a centerline of each of the two divided block groups, is included in a first predetermined angle range (for example, within 30±15 degrees). Note that the first person candidate represents a probability (degree) of a person candidate that is different from the second person candidate, which will be explained later.

[0049] Fig. Fig. 6 is an exemplary diagram for explaining the operation of the person candidate determination unit 164. It is assumed that in a vehicle 1 traveling at night, the Fig. 6A is obtained from the image processing device 120. As can be seen from Fig. 6A, an effective image of a person 208 is a portion below the hips. On this occasion, the grouping unit 162 extracts a block group 210 by grouping a plurality of blocks whose relative positions are close in the range extending from the plane corresponding to the road surface to a predetermined height (for example, 1.0 m) in the distance image 126 according to Fig. 6B, based on the luminance image 124 according to Fig. 6A. The person candidate determination unit 164 creates a virtual frame 212 by enclosing the outer edges of the block group 210 with a rectangle extending in the horizontal and vertical directions. The frame 212 is then divided into two horizontally, and sub-block groups 214a, 214b are created, as shown in Fig. 6C shown.

[0050] Subsequently, the person candidate determination unit 164 refers to all the blocks of each of the sub-block group 214a and sub-block group 214b according to Fig. 6C, and using the least squares method, the person candidate determination unit 164 generates center lines 216a, 216b, as indicated by alternating long and short dashed lines. Since a method for obtaining an approximately straight line (curve) by the least squares method is a well-known technique, a detailed explanation thereof will be omitted here. In the above explanation, a one-dimensional model is used as the center lines 216a, 216b. However, curved lines of two or more degrees may also be used.

[0051] The person candidate determination unit 164 determines whether each of the centerline angles α, β, which are angles between the generated centerlines 216a, 216b and a vertical line 218 extending in the vertical direction in the image (vertical direction), is included within a predetermined first angle range (for example, 30±15 degrees). If both centerline angles α, β are included in the first angle range, the person candidate determination unit 164 assumes that the partial block groups 214a, 214b are two legs of a person and assumes the block group 210 as the first person candidate.

[0052] The first angle range used here may be set to a predetermined angle as described above. However, it may also be changed depending on the distance between the vehicle 1 and the block group 210. The images retrieved in the external environment recognition device 130 as the luminance image 124 and the distance image 126 are expressed with pixels of the same size, regardless of the three-dimensional coordinates. In other words, a block group 210 close to the vehicle 1 (short distance) can be expressed in more detail because the area size thereof is larger compared to the pixel; however, a block group 210 far from the vehicle 1 (long distance) becomes rough or inaccurate because the area size is smaller compared to the pixel.Accordingly, the resolution of the block group 210 is degraded at a distance and it may be impossible to correctly detect the centerline angles between the centerlines 216a, 216b and the vertical line 218.

[0053] Therefore, in accordance with the relative distance from the vehicle 1 in the depth direction z, the person candidate determination unit 164 reduces the first angular range corresponding to the block group 210 located at a distance (applies a stricter condition) compared to the first angular range corresponding to the block group 210 located nearby, so as to avoid erroneously detecting the block group 210 at a distance as the person's legs. This makes it difficult to detect a person even if a person actually exists at a distance. However, the periphery recognition system 100 is hardly affected by such a person located far away, and therefore, this does not cause any problems.

[0054] In this way, the person candidate determination unit 164 can assume the block group 210 grouped by the grouping unit 162 as the first person candidate. However, if the first person candidate is extracted based only on the center lines 216a, 216b of the entire block group 210 divided into two (partial block group 214a, 214b), a specific object other than a person may be incorrectly assumed as the first person candidate.

[0055] Fig. Fig. 7 is an exemplary illustration for explaining the operation of the person candidate determination unit 164. It is assumed that a signage or guidance system 232 is held by a plurality of supports 230, as in Fig. 7. In this case, when the centerline angles α, β between the centerlines 216a, 216b of the plurality of supports 230 and the vertical line 218 are included in the first angle range (for example, 30±15 degrees), the person candidate determining unit 164 can determine that such supports 230 are the first person candidate. Therefore, in the present embodiment, even if the person candidate determining unit 164 once determines the first person candidate, the block group is divided and the first person candidate is specified again with a higher degree of probability of the first person candidate.

[0056] Specifically, the grouping unit 162 further divides the area extending from the plane corresponding to the road surface to the predetermined height in the image into a plurality of areas in the height direction and groups the blocks in each divided area. Then, if a different angle (bend of a joint) formed by center lines of the adjacent (continuous) grouped block groups in the height direction is included in a predetermined second angle range (for example, the absolute value is 5 degrees or more), the person candidate determination unit 164 formally determines that the block group 210 is the first person candidate.

[0057] Fig. Fig. 8 is an exemplary illustration for explaining the further division by the grouping unit 162. For example, the grouping unit 162 groups based on the distance image 126 as shown in Fig. 6B, a plurality of blocks whose relative positions are close into a plurality of areas obtained by further dividing the area extending from the plane corresponding to the road surface to the predetermined height (for example, 1.0 m), and extracts partial block groups 214c, 214d, 214e, 214f, 214g, 214h, such as Fig. 8. Here, the division ratios are, for example, 30 cm, 50 cm, and 20 cm, in this order from the road surface, with the entire area being 1 m. The division ratios are determined based on the assumption that the toes of the legs and the calf part of the person are on the road surface (30 cm), the thighs are located above it in the vertical direction (50 cm), and the hips are located above it in the vertical direction (20 cm).

[0058] Subsequently, the person candidate determination unit 164 determines whether a differential angle formed by sub-block groups adjacent to each other in the height direction, for example, the center lines 216d, 216e of the sub-block group 214d and the sub-block group 214e, is included within the predetermined second angle range (for example, the absolute value is 5 degrees or more), and if it is included, the person candidate determination unit 164 again determines it as the first person candidate. As described above, when considering the bending of a joint that can only be found in a person, the support 230, which is inclined but represented as a simple straight line, may be used as shown in Fig. 7, are excluded from the first person candidate, and this allows the extraction of the first person candidate with a higher degree of accuracy.

[0059] In this case, the division ratios of 30 cm, 50 cm, and 20 cm are chosen for a total length of 1.0 m, based on the ratio of a typical Japanese person's leg and joints. However, the total length and division ratios can be set to any desired values. For example, they can be set differently depending on the country, region, or environment.

[0060] In the extraction processing of the first person candidate arranged with such division ratios, the person candidate determination unit 164 does not perform the determination of the first person candidate with respect to the remaining area unless the difference angle formed by one of the regions in the height direction is included in the predetermined second angle range. This is because, even if a portion cannot be determined as the first person candidate, there is no probability that the block group 210 in question is the first person candidate, and therefore the subsequent process is omitted. As described above, the processing load can be reduced.

[0061] Here, the person candidate determination unit 164 may perform the determination of the first person candidate from the upper side to the lower side in the vertical direction, that is, the person candidate determination unit 164 may perform the determination with respect to the calf and thigh, and then the thigh and hip. This is because the difference angle between the calf and thigh is usually larger than the difference angle between the thigh and hip, and therefore, the determination of the first person candidate can be performed more accurately. First, the person candidate determination unit 164 derives the difference angle between the calf and thigh, and if the value is not included in the second angle range, the difference angle between the thigh and hip is not subsequently derived.Only if the difference angle between the calf and the thigh is included in the second angle range, the difference angle between the thigh and the hip is derived.

[0062] As described above, the first person candidate is determined only when several conditions are met, so although the block group 210 extracted as the first person candidate may be limited, this is more advantageous compared to a case where a block group 210 that should not be detected is erroneously detected as the first person candidate. This is because if the vehicle 1 frequently decelerates in the braking control and the cruise control against the driver's intention due to erroneous detection, comfort during driving may be lost. In the present embodiment, therefore, only a sufficiently reliable block group 210 is extracted as the first person candidate.

[0063] The person candidate tracking unit 166 tracks the block group 210 extracted by the person candidate determination unit 164 as the first person candidate using image processing. This tracking is performed, for example, as follows using images based on image data continuously generated every 1 / 60 of a second. The person candidate tracking unit 166 extracts a block group 210 whose luminance tendency is similar to the block group 210 in the previous image from a position and proximity thereto in the current image corresponding to the position of the block group 210 extracted in the previous image as the first person candidate.The similarity of the block groups 210 is determined by extracting a block determined to have a correlation with the previous block by pattern matching with the current image for each block obtained by further subdividing the block group 210 in the previous image. Various available techniques can be used for tracking processing including this type of similarity determination, and therefore, a detailed description is omitted here.

[0064] The reversal measuring unit 168 measures the number of reversals and the reversal cycle when the person candidate determining unit 164 determines that the block group 210 tracked by the person candidate tracking unit 166 is the first person candidate and is not the first person candidate.

[0065] A person moves forward by alternately moving their right and left legs forward. Therefore, when a walking person is observed from the side, both legs are open, then both legs are closed, and then both legs are open again. Specifically, when a person walks, the following two states are repeated: a state in which both legs are open (a state determined by the person candidate determination unit 164 to be the first candidate) and a closed state (a state determined by the person candidate determination unit 164 not to be the first candidate).Accordingly, in the present embodiment, with respect to the behavior of the block group 210 extracted by the person candidate determination unit 164 as the first person candidate, a determination is made as to whether the block group 210 is the second person candidate that is more likely to be a person, and a determination is made as to whether the first person candidate is moving or not. The first person candidate and the second person candidate are different in the probability of being a person, and both the first person candidate and the second person candidate are candidates of people. In the control in later stages, both are necessary in accordance with the required degree of probability of being a person.

[0066] Fig. 9 is an explanatory diagram for explaining the operation of the inversion measurement unit 168. While a person is walking, both the centerline angle of the right leg and the centerline angle of the left leg are within the first angle range for a period t1 in which, for example, the right leg is in front, and therefore the person candidate determination unit 164 assumes the block group 210 as the first person candidate. However, in a period t2 in which the left leg is moved forward and both legs overlap each other, the centerline angles of both legs are temporarily outside the first angle range. Then, in a period t3 in which the left leg is in front, the centerline angles of both legs are again within the first angle range.As described above, the state in which the block group 210 tracked by the person candidate tracking unit 166 is determined by the person candidate determining unit 164 to be the first person candidate (first person candidate determination), and the state in which the block group 210 tracked by the person candidate tracking unit 166 is determined by the person candidate determining unit 164 not to be the first person candidate (first person candidate non-determination) are expressed in a time chart as shown in FIG. Fig. 9, which shows that its behavior is periodic. The reversal measurement unit 168 measures the number of reversals and the reversal cycle thereof.

[0067] The person candidate specifying unit 170 observes the number of reversals measured by the reversal measuring unit 168, and when the number of reversals is within the predetermined range of numbers (for example, four times or more), the person candidate specifying unit 170 changes the first person candidate to the second person candidate whose probability of being a person is higher than the first person candidate, as indicated by an arrow in Fig. 9 shown.

[0068] In this way, the external environment recognition device 130 can extract one or more block groups as second person candidates from the distance image 126, and the information can be used for various types of control. For example, if any given block group in the detection area is determined to be a person (second person candidate), it is possible to avoid a collision with the specified person and to perform control to maintain a safe distance between the subject vehicle and the person.

[0069] If the reversal cycle measured by the reversal measurement unit 168 is determined to be within a predetermined cycle range (for example, 0.5 Hz or more), the person candidate specifying unit 170 determines that the first person candidate is moving. If the reversal cycle measured by the reversal measurement unit 168 is determined to be outside a predetermined cycle range (for example, less than 0.5 Hz), the person candidate specifying unit 170 determines that the first person candidate is standing, i.e., not moving.

[0070] In this way, the external environment recognition device 130 can determine that the block group 210 extracted as the first person candidate or the second person candidate is moving, and this enables control to avoid collision with the specific person and maintain a safe distance from the person. (External environment detection method)

[0071] In the following, specific operations performed by the external environment detection device 130 will be described with reference to the flowcharts according to the Fig. 10 to 18 explained. Fig. Fig. 10 shows an overall flow of an interrupt process when the image processing device 120 transmits the distance image 126. The Fig. 11 to 18 show individual subroutines therein. In this case, pixels or blocks are used as the target sections of the processing, and the lower left corners of the luminance image 124 and the distance image 126 are the origins. The blocks are used to execute the process according to the external environment recognition method in a range of 1 to 150 blocks in the horizontal direction of the image and 1 to 50 blocks in the vertical direction of the image. In this case, night driving is assumed for the purpose of explanation.

[0072] As in Fig. As shown in FIG. 10, when an interrupt occurs according to the external environment recognition method in response to the reception of a distance image 126 from the image processing device 120, the obtained distance image 126 is referred to, and the position information acquisition unit 160 converts parallax information of each block within the detection range 122 of the distance image 126 into three-dimensional position information (S300). Subsequently, the grouping unit 162 and the person candidate determination unit 164 group a plurality of blocks in the range extending from the plane corresponding to the road surface to the predetermined height and determine whether the grouped block groups are the first person candidate or not (S302).

[0073] Subsequently, the person candidate tracking unit 166 tracks all the block groups 210 extracted by the person candidate determination unit 164 as the first person candidates using image processing (S304). The reversal measurement unit 168 measures the number of reversals and, in parallel, the reversal cycle when the person candidate determination unit 164 determines that the block group 210 tracked by the person candidate tracking unit 166 is the first person candidate and not the first person candidate (S306). Subsequently, based on the result of the reversal measurement unit 168, the person candidate specification unit 170 specifies the second person candidate or determines whether the first person candidate is moving or not (S308). The above operation will be explained in more detail below. (Position information acquisition process S300)

[0074] As in Fig. As shown in Figure 11, the position information acquisition unit 160 initializes a vertical variable j for identifying a block (replacing "0" therein) (S400). Subsequently, the position information acquisition unit 160 adds "1" to the vertical variable j and initializes the horizontal variable i (replacing "0" therein) (S402). Subsequently, the position information acquisition unit 160 adds "1" to the horizontal variable i (S404).

[0075] The position information acquisition unit 160 acquires parallax information dp from the block (i, j, dp) of the distance image 126 (S406). The position information acquisition unit 160 then uses the aforementioned numerical equations 1 to 3 to transform the coordinate of the block (i, j, dp) including the parallax information dp into the coordinate of the point (x, y, z) in the real world, thereby generating the block (i, j, dp, x, y, z) (S408).

[0076] Subsequently, the position information acquisition unit 160 determines whether the horizontal variable i is greater than 150, that is, the maximum value of the horizontal block (S410). If the horizontal variable i is determined to be not greater than the maximum value (NO in S410), the processing is repeated from the increment process of the horizontal variable i in step S404. If the horizontal variable i is determined to be greater than the maximum value (YES in S410), the position information acquisition unit 160 determines whether the vertical variable j is greater than 50, that is, the maximum value of the vertical block (S412). Then, if the vertical variable j is determined to be not greater than the maximum value (NO in S412), the processing is repeated from the increment process of the vertical variable j in step S402.If the vertical variable j is determined to be greater than the maximum value (YES in S412), the position information acquisition process S300 is terminated. As described above, the parallax information dp of the distance image 126 is transformed into three-dimensional position information. (Grouping / Person Candidate Determination Process S302)

[0077] As in Fig. As shown in Figure 12, the grouping unit 162 refers to a predetermined range (for example, 0.5 m) for grouping blocks (S450) and initializes a vertical variable j to specify a block (substituting "0" therein) (S452). Subsequently, the grouping unit 162 adds "1" to the vertical variable j and initializes (sets "0") the horizontal variable i (S454). Subsequently, the grouping unit 162 adds "1" to the horizontal variable i (S456).

[0078] The grouping unit 162 obtains a block (i, j, dp, x, y, z) from the distance image 126 (S458). Then, a determination is made as to whether the height y of the block (i, j, dp, x, y, z) is included in the boundary range, that is, a range extending from the plane corresponding to the road surface to the predetermined height (e.g., 1.0 m) (S460). In this case, when the height y is determined to be within the boundary area (YES in S460), the grouping unit 162 determines, based on a coordinate (x, y, z) in the real world corresponding to the block (i, j, dp, x, y ,z), whether there is another block (i, j, dp, x, y ,z) that is within the predetermined area and that is in the boundary area (S462).

[0079] If another block exists (YES in S462), the grouping unit 162 determines whether a first group number g is attached to any or all of the blocks in the predetermined range, including the block currently being examined (S464). If the first group number g is given to any of them (YES in S464), the grouping unit 162 assigns a value to all of the blocks located in the predetermined range and all of the blocks given the same first group number g, the value being a smaller one of the smallest first group number g among the first group numbers assigned thereto or the smallest value of numbers not yet used as the first group number, so as to be expressed as one block (i, j, p, dp, x, y, z, g) (S466).If the first group number g is not given to any of them (NO in S464), the smallest value of the numbers that have not been used yet is newly assigned as a first group number to all the blocks located within the predetermined range, including the block under investigation (S468).

[0080] In this way, when a plurality of blocks are located within the predetermined range, the grouping process is performed by assigning a first group number g. Here, if the first group number g is determined not to have been given to any of the plurality of blocks, a new first group number g is assigned, and if the first group number g is determined to have already been given to one of those, the same first group number g is assigned to the other blocks. However, if there are a plurality of first group numbers g in the plurality of blocks, the first group numbers g of all the blocks are replaced by the one first group number g, so as to treat the blocks as one group.

[0081] In the above description, the first group numbers g of not only all the blocks within the predetermined range but also all the blocks assigned the same first group number g are changed at one time. The main reason for this is to prevent the group that has already been unified by changing the first group numbers g from being divided. Furthermore, a smaller number than the smallest first group number g or the smallest value of numbers that have not yet been used is used as the first group number to prevent a number from being skipped as much as possible during group numbering. This prevents the maximum value of the first group number g from becoming unnecessarily large, and the processing load can be reduced.

[0082] Back on Fig. 12, the processing in a subsequent step S470 is performed when the height y is outside the limitation range (NO in S460) or when no other block exists in the predetermined range (NO in S462).

[0083] Next, the grouping unit 162 determines whether the horizontal variable i is greater than 150, which is the maximum value of the number of pixels in the horizontal direction, or not (S470). If the horizontal variable i is determined to be not greater than the maximum value (NO in S470), the processes are repeated from the increment process of the horizontal variable i in step S456. If the horizontal variable i is determined to be greater than the maximum value (YES in S470), the grouping unit 162 determines whether the vertical variable j is greater than 50, which is the maximum value of the number of pixels in the vertical direction, or not (S472). If the vertical variable j is determined to be not greater than the maximum value (NO in S472), the processes are repeated from the increment process of the vertical variable j in step S454.If the vertical variable j is determined to be greater than the maximum value (YES in S472), the subsequent step is performed.

[0084] As in Fig. As shown in Fig. 13, the person candidate determination unit 164 refers to the smallest first group number g and sets the block group assigned the first group number g (all blocks with the first group number g) (S500). Subsequently, the person candidate determination unit 164 determines whether the size of the area of ​​the block group thus set is equal to or larger than a predetermined size of an area that can be regarded as a person's leg (S502). If the condition here is satisfied (YES in S502), the block group is divided into two in the horizontal direction of the image (S504), and the center line of each of the two divided partial block groups is derived (S506).

[0085] Subsequently, the person candidate determination unit 164 determines whether the centerline angle of the centerline of each of the two divided block groups is included in the first predetermined angle range (for example, within 30±15 degrees) (S508). If the centerline angle is included in the first angle range (YES in S508), the block group in question is temporarily determined to be the first person candidate, and a person candidate flag (flag) f="1" is assigned to the blocks (S510). The first angle range can be changed according to the distance between the subject vehicle 1 and the block group.

[0086] When the size range of the block group does not satisfy the condition (NO in S502) or when the center line angle is not included in the first angle range (NO in S508), the person candidate determination unit 164 assigns a person candidate flag f=“0” to the blocks (S512) and the process flow in the subsequent step S514 is performed.

[0087] Then, the person candidate determination unit 164 determines whether the process in steps S502 to S512 has been performed for all of the multiple block groups (S514). If the process flow in steps S502 to S512 has not been performed for all block groups (NO in S514), the first group number g is updated and a new block group is set (S516), and then the process flow in step S502 and the subsequent steps is repeated for the new block group. On the other hand, if the process flow in steps S502 to S512 is determined to have been performed for all block groups (YES in S514), a subsequent step is performed.

[0088] As in Fig. As shown in Figure 14, the grouping unit 162 refers to the predetermined area for grouping the blocks (for example, 0.5 m) and the division ratios of a plurality of areas for dividing in the height direction (S550), and initializes the vertical variable j for specifying the blocks (replacing "0" herein) (S552). Subsequently, the grouping unit 162 adds "1" to the vertical variable j and initializes the horizontal variable i (replacing "0" herein) (S554). Subsequently, the grouping unit 162 adds "1" to the horizontal variable i (S556).

[0089] The grouping unit 162 obtains the block (i, j, dp, x, y, z, g, f) from the distance image 126 (S558). Then, a determination is made as to whether or not the person candidate flag f of the block (i, j, dp, x, y, z, g, f) is equal to "1" (S560). In this case, if the person candidate flag f is determined to be "1" (YES in S560), the grouping unit 162 refers to the division ratios and deduces which area of ​​the multiple areas divided in the height direction the block (i, j, dp, x, y, z, g, f) belongs to, and obtains the height range of the area (S562).

[0090] Subsequently, the grouping unit 162 determines, based on a coordinate (x, y, z) in the real world corresponding to the block (i, j, dp, x, y, z), whether there is another block (i, j, dp, x, y, z, g, f) that is in the predetermined range and whose height y is included in the height range (S564).

[0091] If another block (i, j, dp, x, y, z, g, f) exists (YES in S564), the grouping unit 162 determines whether or not a second group number gg is attached to any one of all the blocks in the predetermined range, including the block currently being examined (S566). If the second group number gg is assigned to any one of them (YES in S566), the grouping unit 162 assigns a value to all the blocks included in the predetermined range and all the blocks given the same second group number gg, the value being a smaller one of the smallest second group number gg among the second group numbers assigned thereto, or the smallest value of the numbers not yet used as a second group number, so as to express as one block (i, j, dp, x, y, z, g, f, gg) (S568).If the second group number gg has not been assigned to any of them (NO in S566), the smallest value of numbers that have not yet been used as a second group number is newly assigned to all the blocks within the predetermined range including the block under investigation (S570).

[0092] In this manner, as with the first group number g, the grouping process is performed by assigning a second group number gg when there are a plurality of blocks within the predetermined range. On this occasion, if the second group number gg is determined not to have been assigned to any of the plurality of blocks, a new second group number gg is assigned, and if the second group number gg is determined to have already been assigned to one of them, the same second group number gg is assigned to the other blocks. However, if a plurality of second group numbers gg already exist in the plurality of blocks, the second group numbers gg of all the blocks are replaced by the one second group number gg so as to treat the blocks as one group.

[0093] When the person candidate flag f is not "1" (NO in S560), or when no other block is in the predetermined area (NO in S564), the process in the subsequent step S572 is performed.

[0094] Next, the grouping unit 162 determines whether the horizontal variable i is greater than 150, which is the maximum value of pixel numbers in the horizontal direction (S572). If the horizontal variable i is determined not to be greater than the maximum value (NO in S572), the processes are repeated from the increment process of the horizontal variable i in step S556. If the horizontal variable i is determined to be greater than the maximum value (YES in S572), the grouping unit 162 determines whether the vertical variable j is greater than 50, which is the maximum value of pixel numbers in the vertical direction (S574). If the vertical variable j is determined not to be greater than the maximum value (NO in S574), the process is repeated from the increment process of the vertical variable j in step S554.If the vertical variable j is determined to be greater than the maximum value (YES in S574), the following step is performed.

[0095] As in Fig. As shown in FIG. 15, the person candidate determination unit 164 refers to the smallest first group number g and sets the block group to which the first group number g is appended (S600). Subsequently, the person candidate determination unit 164 determines whether the person candidate flag f of the block (i, j, dp, x, y, z, g, f, gg) in the block group is "1" or not (S602). On this occasion, if the person candidate flag f is determined to be "1" (YES in S602), all the sub-block groups belonging to the block group that has been set (sub-block groups whose first group numbers g are the same but which have different second group numbers gg) are retrieved and divided in the horizontal direction of the image (S604). Then, the person candidate determination unit 164 derives all the center lines of the two divided sub-block groups (S606).

[0096] Subsequently, the person candidate determination unit 164 extracts combinations of sub-block groups adjacent to each other in the height direction from a plurality of sub-block groups (six sub-block groups in this case) and determines whether the difference angles formed by the center lines of the combinations of extracted sub-block groups are included in the predetermined second angle range (for example, the absolute value is 5 degrees or more) (S608). If the difference angle is determined to be included in the second angle range (YES in S608), the block group in question is determined to be the first person candidate, and the person candidate flag f="1" is maintained in the blocks (S610).

[0097] The person candidate determination unit 164 then determines whether the processes in steps S608, S610 have been performed for all combinations of sub-block groups (S612). Here, if all combinations of sub-block groups are determined to be unfinished (NO in S612), the second group number gg is updated, and new combinations of sub-block groups that are adjacent to each other are set (S614), and the processes of step S608 and subsequent steps are repeated with respect to the new combinations of sub-block groups. On the other hand, if all combinations of sub-block groups are determined to be completed (YES in S612), step S618 is subsequently performed.

[0098] If the difference angle is determined not to be within the second angle range (NO in S608), the person candidate determination unit 164 replaces the person candidate flag f of the block with "0" (S616) and terminates the processing with respect to the block group, and then the processing in step S618 is subsequently performed. If the person candidate flag f is determined not to be "1" (NO in S602), the processing in step S618 is also performed.

[0099] The person candidate determination unit 164 determines whether the process flow in steps S602 to S616 is performed for all of the multiple block groups (S618). If the process flow in steps S602 to S616 is determined not to be performed for all block groups (NO in S618), the first group number g is updated and a new block group is set (S620), and then the process flow in step S602 and subsequent steps is repeated for the new block group. On the other hand, if the process flow in steps S602 to S616 is determined to be performed for all block groups (YES in S618), the grouping / person candidate determination process S302 is terminated. In this way, the first person candidate recognizable as a person can be extracted with a high degree of accuracy. (Personal candidate tracking process S304)

[0100] As in Fig. As shown in Fig. 16, the person candidate tracking unit 166 sets a registered block group (S650) and determines whether or not there is a block group, the luminance tendency of which is equal to the luminance of the previous image in the block group, at a position and proximity thereto in the current image corresponding to the previous position of the block group that was set (S652). If there is a block group that satisfies the condition (YES in S652), the block group of the current image is updated as the previous image to continue tracking the block group (S654). If there is no block group that satisfies the condition (NO in S652), tracking the block group has failed, and the registration of the tracked block group is deleted (S656).

[0101] Next, the person candidate tracking unit 166 determines whether the process flow in steps S652 to S656 has been executed for all registered multiple block groups (S658). If it is determined that the process flow in steps S652 to S656 has not been executed for all registered block groups (NO in S658), a new registered block group is set (S660), and step S652 and the subsequent steps are repeated for the new block group. On the other hand, if it is determined that the process flow in steps S652 to S656 has been executed for all block groups (YES in S658), the process flow in step S662 is executed.

[0102] Subsequently, the person candidate tracking unit 166 determines whether or not there is a block group newly determined by the person candidate determination unit 164 to be the first person candidate (S662). If the person candidate tracking unit 166 determines that there is such a block group (YES in S662), the block group is newly registered as a tracking target (S664).

[0103] Here, in association with the block group, the following variables are set as variables used at a later stage in the reversal measurement process S306: the number of reversals N = "1" and the reversal flag = "1." A reversal time counter associated with this block is provided and reset to zero. In this case, the reversal time counter is a means for measuring the time from the reset, that is, the reversal operation. Thereafter, the person candidate tracking process S304 is terminated. In this way, the block group, which itself was once determined to be the first person candidate, can be tracked while existing in the image, regardless of movement of the block group. (Reverse measurement process S306)

[0104] As in Fig. As shown in Fig. 17, the reversal measuring unit 168 sets a registered block group (S700) and determines whether or not this block group is determined by the person candidate determining unit 164 to be the first person candidate and whether or not the reversal flag is zero (S702). This indicates a reversal from a state where it is determined not to be the first person candidate to a state where it is determined to be the first person candidate.

[0105] The person candidate determination unit 164 determines that the block group is the first person candidate and that the reversal flag is zero (YES in S702). The reversal measuring unit 168 increments the number of reversals N by one (S704) and reads the time from the reversal time measuring or counter (S706). Then, the reversal measuring unit 168 assumes a reciprocal of the time read from the reversal time counter as one reversal cycle and resets the reversal time counter (S708). When a reversal occurs from a state in which it is determined by the person candidate determination unit 164 not to be the first person candidate to a state in which it is determined by the person candidate determination unit 164 to be the first person candidate, the number of reversals and the reversal cycle can be derived in this way. In this case, the reversal cycle is derived based on one reversal.Alternatively, an average value can be derived based on several reversals.

[0106] Subsequently, the reversal measurement unit 168 sets the reversal flag so that the reversal flag reflects the result indicating whether or not the person candidate determination unit 164 determines the block group as the first person candidate (S710). Specifically, if the person candidate determination unit 164 determines that the block group is the first person candidate, the reversal measurement unit 168 sets the reversal flag to "1." If the person candidate determination unit 164 determines that the block group is not the first person candidate, the reversal measurement unit 168 sets the reversal flag to "0." In the subsequent reversal measurement process S306, the reversal flag is used as a previous value indicating whether the block group is determined to be the first person candidate.

[0107] Subsequently, the inverse measurement unit 168 determines whether the processes in steps S702 to S710 have been executed for all registered multiple block groups (S712). If it is determined that the process flow in steps S702 to S710 has not been executed for all registered block groups (NO in S712), a new registered block group is set (S714), and step S702 and subsequent steps are repeated for the new block group. On the other hand, if the process flow in steps S702 to S710 is determined to have been executed for all registered block groups (YES in S712), the inverse measurement process S306 is terminated. (Person candidate specification process S308)

[0108] As in Fig. As shown in FIG. 18, the person candidate specifying unit 170 sets a registered block group (S750) and determines whether or not the number of reversals of this block group is included in a predetermined frequency range (for example, four times or more) (S752). If the number of reversals is determined to be included in the frequency range (YES in S752), the person candidate flags f for all the blocks (i, j, dp, x, y, z, g, f, gg) constituting the block group are changed from "1" to "2" (S754). In this way, the block group can be specified as the second person candidate whose probability of being a person is higher than that of the first person candidate.

[0109] Subsequently, the person candidate specifying unit 170 determines whether or not the reverse cycle is included within the predetermined cycle range (for example, 0.5 Hz or more) (S756). If the reverse cycle is determined to be included within the cycle range (YES in S756), the person candidate specifying unit 170 determines that the block group in question is moving (S758). If the reverse cycle is determined not to be within the cycle range (NO in S756), the person candidate specifying unit 170 determines that the block group is stationary (S760).

[0110] Subsequently, the person candidate specifying unit 170 determines whether or not the process flow in steps S752 to S760 has been executed for all registered multiple block groups (S762). At this time, if it is determined that the process flow in steps S752 to S760 has not been executed for all registered block groups (NO in S762), a new registered block group is set (S764), and step S752 and the subsequent steps are repeated for the new block group. On the other hand, if the process flow in steps S752 to S760 is determined to have been executed for all block groups (YES in S762), the person candidate specifying process S308 is terminated. Thus, the second person candidate can be extracted, and a determination can be made as to whether the block group is moving.

[0111] As described above, according to the external environment recognition device 130 and the external environment recognition method, the efficiency of specifying a person and the accuracy of specifying a person can be improved by directly specifying the person with a lower portion area (the lower half of the body) of the person.

[0112] In addition, a program that enables a computer to perform the function of the external environment detection device 130 is provided, as well as a storage medium such as a computer-readable floppy disk, a magneto-optical disk, a ROM, a CD, a DVD, or a BD (Blu-ray Disc) for storing the program. Here, a program means a data processing function written or described in any language or description method. (Modifications)

[0113] In the above embodiment, the relative positional relationship is used as the first relative relationship between blocks used by the grouping unit 162 for the grouping process. However, the first relative relationship is not limited to this case. For example, a difference in luminances and continuity in the edge direction can also be used. The difference in luminances and continuity in the edge direction will be briefly explained below. (Difference of luminances)

[0114] The grouping unit 162 may use the luminance image 124 instead of the distance image 126 to group multiple blocks in the luminance image 124 based on a difference in luminances, thereby generating block groups that serve as section groups. For example, the grouping unit 162 performs the grouping process in such a way that any given block is used as a base point, an adjacent block is retrieved, and a difference in the luminances of the two blocks is derived. A block whose difference in luminances is within a predetermined range (for example, the difference of each RGB within 1 / 10 of the full scale) is assumed to belong to the same target object. (Continuity in edge direction)

[0115] The Fig. 19A to 19G are exemplary illustrations for explaining the continuity in the edge direction. Among these, Fig. 19A shows a simplified video image of the lower half of a person's body 208 in the luminance image 124, taken at night. A luminance distribution as shown in Fig. 19B is shown by enlarging a given block 240a in Fig. 19A. A luminance distribution as shown in Fig. 19C is obtained by enlarging another block 240b. The range of the luminance range extends from 0 to 255. In Fig. 19B, assume that a white field represents a luminance of "200" and a black field represents a luminance of "0." In this case, in the block figure, assume that the luminance of the upper left pixel is defined as A, the luminance of the upper right pixel is defined as B, the luminance of the lower left pixel is defined as C, and the luminance of the lower right pixel is defined as D, where the horizontal direction component in the edge direction is defined as (A+B) - (C+D), and the vertical direction component in the edge region is defined as (A+C) - (B+D).

[0116] Accordingly, the horizontal direction component in the edge direction of the block 240a, as in Fig. 19B, (A+B) - (C+D) = (200+0) - (200+0) = 0, and the vertical direction component in the edge direction thereof is (A+C) - (B+D) = (200+200) - (0+0) = +400. Therefore, the horizontal direction component is "0" and the vertical direction component is "+400." The edge direction is denoted as an upward arrow in the vertical direction, as in Fig. 19D. However, the horizontal component is, as shown in Fig. 19F, positive in the right direction, while the vertical component is positive in the upward direction.

[0117] Similarly, the horizontal direction component in the edge direction of the block 240a, as in Fig. 19C, (A+B) - (C+D) = (0+0) - (200+200) = -400, and the vertical direction component in the edge direction thereof is (A+C) - (B+D) = (0+200) - (0+200) = 0. Therefore, the horizontal direction component is "-400" and the vertical direction component is "0." The edge direction is denoted as a left arrow in the horizontal direction, as shown in Fig. 19E shown.

[0118] As previously described, half of the area in the block is subtracted from the other half of the block. This configuration allows for noise reduction and offsetting of the luminance trapped within the entire block, thus allowing for proper edge extraction. Furthermore, the computational burden is low because the edge direction can be derived using a simple calculation of only addition and subtraction.

[0119] The continuity in the edge direction serves to determine whether adjacent blocks are equal in the derived edge direction and to assign these blocks to each other. In this modification, both the horizontal and vertical direction components are defined using a unit length, which simplifies the variation in the edge direction. More precisely, both the horizontal and vertical direction components are assumed to be either -1, 0, or +1. In this case, as in Fig. As shown in Figure 19G, the edge direction can be limited to nine states, including eight directions, each different by 45 degrees, and a zero direction in which both the horizontal direction component and the vertical direction component are zero. This allows the computational load to be greatly reduced by the grouping unit 162.

[0120] In this modification, the grouping unit 162 derives the edge direction for each block of the image in which the lower half of the person's body is displayed, and associates adjacent blocks whose edge directions are the same with each other, thus generating continuous edge traces. In this way, the shape of the lower half of the person's body can be specified, as well as the difference in luminance and the relative positional relationship.

[0121] Although a preferred embodiment of the present invention has been described herein with reference to the accompanying drawings, it should be understood that the present invention is not limited to such an embodiment. It will be understood by those skilled in the art that various modifications may be made without departing from the scope of the invention.

[0122] For example, in the above embodiment, the angle formed by the vertical line and the center line of each of the two divided groups (center line angle) is used as the second relative relationship for determining the first person candidate. However, the embodiment is not limited to this. One or more relationships selected from the group including the shape, the area size, the symmetry, and the luminance may be used. For example, the first person candidate is determined when the similarity in shape or a difference in the area size is within a predetermined range, when there is a high degree of symmetry in the horizontal direction, or when the distribution of the luminance or a difference in the total amount is within a predetermined range.

[0123] In the above embodiment, the three-dimensional position of the target object is derived based on the parallax between image data using the plurality of image pickup devices 110. However, the present invention is not limited to such a case. For example, various known distance measuring devices, such as a laser radar distance measuring device, could alternatively be used. In this case, the laser radar distance measuring device emits a laser beam toward the detection area 122, receives the reflected light when the laser beam is irradiated onto the object, and measures the distance to the object based on this required time.

[0124] In the above embodiment, for example, the position information acquisition unit 160 receives the distance image (parallax information) 126 from the image processing device 120 and generates the three-dimensional position information. However, the present invention is not limited to such a case. The image processing device 120 may generate the three-dimensional position information in advance, and the position information acquisition unit 160 may obtain the generated three-dimensional position information. Such functional distribution can reduce the processing load of the external environment recognition device 130.

[0125] In the above embodiment, the position information acquisition unit 160, the grouping unit 162, the person candidate determination unit 164, the person candidate tracking unit 166, the reverse measurement unit 168, and the person candidate specification unit 170 are configured to function with software by means of the central control unit 156. However, these functional units may also be configured with hardware.

[0126] The steps of the environment detection method in this description do not necessarily have to be processed chronologically in the order described in the flowchart. The steps may be performed in parallel or may involve processing via subroutines.

[0127] The present invention can be applied to the external environment recognition device and the external environment recognition method that recognizes the environment outside the subject vehicle. SUMMARY OF REVELATION

[0128] An environment recognition device and an environment recognition method are provided. An external environment recognition device 130 acquires an image in a detection area, generates a block group by grouping, based on a first relative relationship between blocks, a plurality of blocks in an area extending from a plane corresponding to the road surface to a predetermined height in the acquired image, divides the block group into two in a horizontal direction of the image, and determines whether the block group is a first person candidate based on a second relative relationship between two divided block groups.

Claims

[1] An external environment detection device (130) comprising: an image acquiring unit for acquiring an image in a detection area (122); a grouping unit (162) for generating a section group by grouping, based on a first relative relationship between sections, a plurality of sections in a range extending from a plane corresponding to a road surface to a predetermined height in the acquired image; and a person candidate determining unit (164) for dividing the section group into two in a horizontal direction of the image and determining, based on a second relative relationship between two divided section groups, whether the section group is a first person candidate which is a candidate of a person (208), wherein the second relative relationship is whether or not each centerline angle formed by a vertical line (218) and a centerline (216a, 216b) of each of the two divided section groups is included in a predetermined first angular range. [2] The external environment recognition device (130) according to claim 1, wherein the first relative relationship is one of a group including a relative positional relationship, a difference in luminance, or a continuity in an edge direction. [3] The external environment detection device (130) according to any one of claims 1 to 2, wherein the grouping unit (162) further divides, into a plurality of areas in a height direction, the area extending from the plane corresponding to the road surface to the predetermined height in the image, and groups a plurality of sections in each of the divided areas, and the person candidate determining unit (164) determines the first person candidate when the second relative relationship between the two divided section groups is satisfied and a difference angle between center lines (216a, 216b) of the grouped section groups that are adjacent to each other in the height direction is equal to or more than a predetermined second angle range. [4] The external environment recognition device (130) according to claim 3, wherein, when a difference angle formed by one of the areas in the height direction is determined not to be equal to or larger than the predetermined second angle range, the person candidate determination unit (164) does not perform the determination of the first person candidate with respect to the remaining area. [5] The external environment recognition device (130) according to any one of claims 1 to 4, wherein the person candidate determination unit (164) changes the first angular range according to a relative distance between a subject vehicle and the section group. [6] The external environment detection device (130) according to any one of claims 1 to 5, further comprising: a person candidate tracking unit (166) for tracking the section group determined by the person candidate determining unit (164) to be the first person candidate; a reversal measuring unit (168) for measuring a number of reversals when the person candidate determining unit (164) determines that the section group tracked by the person candidate tracking unit (166) is the first person candidate and the first person candidate has not been; and a person candidate specifying unit (170) for specifying the first person candidate as a second person candidate when the number of reversals is determined to be equal to or greater than the predetermined frequency range. [7] The external environment detection device (130) according to any one of claims 1 to 5, further comprising: a person candidate tracking unit (166) for tracking the section group determined by the person candidate determining unit (164) to be the first person candidate; a reversal measuring unit (168) for measuring a reversal cycle when the person candidate determining unit (164) determines that the section group tracked by the person candidate tracking unit (166) is the first person candidate and the first person candidate has not been; and a person candidate specifying unit (170) for determining that the first person candidate is moving when a reversal cycle is determined to be equal to or more than a predetermined cycle range. [8] An external environment detection method comprising the steps of: Obtaining an image in a detection area (122); Creating a section group by grouping, based on a first relative relationship between sections, a plurality of sections in a range extending from a plane corresponding to the road surface to a predetermined height in the acquired image; Dividing the section group into two in a horizontal direction of the image and Determining, based on a second relative relationship between two divided section groups, whether the section group is a first person candidate that is a candidate of a person (208), wherein the second relative relationship is whether or not each centerline angle formed by a vertical line (218) and a centerline (216a, 216b) of each of the two divided section groups is included in a predetermined first angular range.

Images