External recognition device and external recognition method

Abstract

An external recognition device is provided with: a texture region division unit that divides external information into a plurality of texture regions for each texture specified on the basis of the external information acquired from an external information acquisition unit; a parallax generation unit that generates a parallax corresponding to an external depth distance on the basis of the external information; a parallax voting unit that votes for a parallax corresponding to the depth distance of the object determined on the basis of the external information in a voting space having an external depth distance on one axis; an additional parallax voting unit that votes for the additional parallax on the basis of the texture region in the voting space in which the parallax has been voted; and an object detection unit that recognizes the outside by detecting an object on the basis of the number of parallax votes voted in the voting space.

Description

Technical Field

[0001] The invention relates to an external identification device and an external identification method. Background Art

[0002] In the implementation of autonomous driving and advanced driving assistance systems, the importance of external recognition devices that monitor the outside of the vehicle and detect obstacles in the driving path, lane information on the driving path, and other objects necessary for the vehicle to travel and recognize the outside is increasing. In order to improve the detection performance of objects, multiple cameras (stereo cameras) are installed on the vehicle to obtain information about the vehicle's surroundings captured by each camera, thereby realizing the function of recognizing the surrounding conditions of the vehicle.

[0003] The stereo camera used in the external recognition device mounted on the vehicle is a device in which two cameras are arranged on the vehicle at a predetermined interval in the horizontal direction. In addition, the external recognition device uses the parallax of the overlapping area of ​​the images captured by the two cameras to measure the distance from the camera to the captured object, thereby calculating the possibility of contact with the object located in the direction of vehicle travel. However, there are various objects outside, and the information required for the vehicle to drive automatically is a lot. Therefore, a technology that can accurately recognize the outside of the vehicle is required.

[0004] Patent document 1 states that "the calculation unit further performs object detection based on the visual information, and the control unit controls the moving body so that the detected object is projected at a specified position in the visual information. In addition, the holding unit holds an object model for object detection and holds a target position posture relative to the object, which is the position posture that the AGV should be in relative to the object when it reaches its destination." Prior art literature Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-125345 Summary of the invention Problem that the invention aims to solve

[0006] The parallax image formed by the parallax obtained from two images can be assumed to be a parallax image formed by a three-dimensional point cloud. In previous external recognition devices, it is difficult to find objects that are different objects but have no difference in height or a low height difference based on the parallax image. For example, if the step difference at the boundary between the lane and the sidewalk or the central dividing strip is low, the previous external recognition device can hardly detect the parallax at the boundary, so it is impossible to correctly detect the boundary based on the parallax image. Similarly, for objects with uniform texture (such as black cars) or objects photographed at night, it is difficult for previous external recognition devices to detect the parallax of the object, and the difference between the object and the background easily becomes unclear, so the object cannot be detected correctly. In addition, even if the object has a pattern on the texture, if the height of the object is low, the previous external recognition device cannot detect the object.

[0007] The technology disclosed in Patent Document 1 requires a specific object model to be maintained in advance, and cannot correctly detect objects in an environment such as an outdoor lane where there are various objects not specified in the object model. In addition, if it is a boundary of a road with few steps or an object with little texture difference (such as a black vehicle), it is more difficult to detect the object, and the technology disclosed in Patent Document 1 cannot be used for a control system based on automatic driving.

[0008] The present invention has been made in view of such circumstances, and an object of the present invention is to enable detection of various objects that are difficult to detect using information obtained only from parallax. Technical means of solving problems

[0009] The external recognition device of the present invention comprises: a texture area segmentation unit, which segments the external information into multiple texture areas according to the texture determined based on the external information obtained from the external information acquisition unit; a disparity generation unit, which generates a disparity corresponding to the external depth distance based on the external information; a disparity voting unit, which votes for the disparity corresponding to the depth distance of the object determined based on the external information in a voting space having the external depth distance on one axis; an additional disparity voting unit, which votes for the additional disparity based on the texture area in the voting space that has voted for the disparity; and an object detection unit, which detects the object based on the number of votes for the parallax voted in the voting space, thereby recognizing the outside. Effects of the Invention

[0010] According to the present invention, by voting for an additional disparity based on a texture area in a voting space that has voted for disparity, it is possible to accurately detect an object. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a block diagram showing an example of the internal configuration of an external recognition device according to one embodiment of the present invention. Figure 2This is a block diagram showing an example of the hardware configuration of a computer according to one embodiment of the present invention. Figure 3 This is a diagram showing an example of frequency distribution of distances in a region (road surface) where no three-dimensional object exists according to one embodiment of the present invention. Figure 4 This is a diagram showing an example of frequency distribution of distances in a region where a solid object (front vehicle) exists according to one embodiment of the present invention. Figure 5 FIG. 1 is a diagram showing an example of a parallax image and a UD voting space according to an embodiment of the present invention. Figure 6 1 is a diagram showing an example of an original image captured by a stereo camera and a UD voting space according to an embodiment of the present invention. Figure 7 This is a block diagram showing an example of the internal structure of a texture region segmentation unit according to one embodiment of the present invention. Figure 8 This is a block diagram showing an example of the internal structure of an additional disparity voting unit according to one embodiment of the present invention. Fig. 9 FIG. 1 is a diagram showing transitions of images for explaining semantic area segmentation according to an embodiment of the present invention. Fig.10 This is a diagram showing examples of voting results in each image according to one embodiment of the present invention. Fig.11 This is a diagram showing an example of performing obstacle interpolation voting after semantic area segmentation according to one embodiment of the present invention. Fig.12 This is a diagram showing examples of voting results in each image according to one embodiment of the present invention. Fig.13 FIG. 1 is a diagram showing an example of performing obstacle interpolation voting after performing the individual object region segmentation according to one embodiment of the present invention. Fig.14 This is a diagram for explaining a process of recognizing a traffic cone according to one embodiment of the present invention. Fig.15 This is a diagram for explaining processing on a plurality of texture regions included in the same object according to one embodiment of the present invention. Fig.16 This is a diagram for explaining a process of confirming that a single object region has a size within a predetermined range according to one embodiment of the present invention. Fig.17 This is a flowchart showing an example of processing of the external recognition device according to one embodiment of the present invention. DETAILED DESCRIPTION

[0012] Hereinafter, the mode for implementing the present invention will be described with reference to the drawings. In this specification and the drawings, components having substantially the same function or configuration are denoted by the same reference numerals, and repeated descriptions are omitted.

[0013] [One embodiment] First, refer to Figure 1 An example of the internal configuration of the external recognition device 1 according to one embodiment of the present invention will be described. Figure 1 : is a block diagram showing an example of the internal structure of the external recognition device 1. The external recognition device 1 is a part of an electronic control unit (ECU: Electronic Control Unit) mounted on a vehicle, and is a device that recognizes external objects based on external information obtained from the direction of travel of the vehicle. In addition, the vehicle equipped with the external recognition device 1 is also referred to as the "vehicle". For example, the present invention can be applied to a computing device for vehicle control that can communicate with an on-board ECU (Electronic Control Unit) for an advanced driver assistance system (ADAS: Advanced Driver Assistance System) or autonomous driving (AD: Autonomous Driving).

[0014] The external recognition device 1 includes an external information acquisition unit 100 , a disparity generation unit 200 , a disparity voting unit 300 , a texture region segmentation unit 400 , an additional disparity voting unit 500 , an object detection unit 600 , and an alarm control unit 700 .

[0015] The external information acquisition section (external information acquisition section 100) acquires information detected by the sensing device and information detected by other sensing devices as external information. The sensing device and other sensing devices are provided in the traveling direction of the vehicle (the front and rear of the vehicle).

[0016] It is assumed that the sensing device is a first camera, and the other sensing device is a second camera arranged in the horizontal direction of the first camera. In this case, the first camera and the second camera are used as stereo cameras, and the external information acquisition unit (external information acquisition unit 100) acquires the external image captured by the first camera and the second camera as external information. It can also be assumed that the sensing device is a monocular camera, and the other sensing device is a measuring device (LiDAR (Light Detection and Ranging), millimeter wave radar, infrared sensor, etc.) that measures the distance to the object by irradiating electromagnetic waves (laser, millimeter wave, infrared, etc.) to obtain the distribution of the three-dimensional point group. In this case, the external information acquisition unit (external information acquisition unit 100) acquires the external image captured by the monocular camera and the distribution of the three-dimensional point group output from the measuring device as external information.

[0017] The parallax generation unit (parallax generation unit 200) generates a parallax corresponding to the external depth distance based on the external information input from the external information acquisition unit 100. For example, the parallax generation unit 200 performs pattern matching on two images simultaneously captured by a stereo camera, thereby generating a parallax of an arbitrary object in the image. The parallax generated by the parallax generation unit 200 is described later. Figure 3 and Figure 4 10. The parallax image 10 includes distance information indicating the relative distance from the stereo camera to the object in the depth direction. In the following description, the relative distance from the stereo camera to the object in the depth direction in the parallax image 10 is referred to as "distance".

[0018] In a UD voting space having an external depth distance on one axis, the parallax voting unit (parallax voting unit 300) votes for the parallax corresponding to the depth distance of the object determined based on external information. Therefore, the parallax voting unit 300 calculates the distance from the vehicle to the object based on the parallax input from the parallax generating unit 200, and votes for the parallax in the UD voting space based on the distance. The UD voting space is a space in which the horizontal axis of the image coordinate is set to U and the vertical axis is set to the parallax value D. The parallax value D represents the depth distance. For example, if the external information indicates that there is another vehicle 20m ahead in the direction of travel of the vehicle, an image indicating the presence of the other vehicle is displayed in the parallax image. The details of the UD voting space will be described later. Figure 3 to Figure 6 Provide explanation.

[0019] The texture region segmentation unit (texture region segmentation unit 400) segments the external information into a plurality of texture regions according to the texture determined based on the external information, and the external information is acquired from the external information acquisition unit (external information acquisition unit 100). For example, the texture region segmentation unit 400 performs region segmentation according to the lanes, sidewalks, lanes, and vehicles that have the same texture among the various objects reflected in the external information. Texture refers to the shape, pattern, color, etc. of the surface of an object. Therefore, even the same vehicle or lane is sometimes divided into different texture regions if the texture is different. The internal structure of the texture region segmentation unit 400 is, for example, as described later. Figure 7 shown.

[0020] The additional disparity voting unit (additional disparity voting unit 500) votes for the additional disparity according to the texture area divided by the texture area division unit 400 for the UD voting space for which the disparity is voted by the disparity voting unit 300. In addition, the additional disparity voting unit (additional disparity voting unit 500) votes for the weighted disparity according to the texture area. Therefore, the portion corresponding to the boundary of the divided texture area in the UD voting space is emphasized (see the following description). Fig. 9). For example, the additional parallax voting unit 500 votes for the additional parallax by weighting low-height objects such as a road edge without a step, thereby enabling the object detection unit 600 to detect the road edge without a step.

[0021] In addition, the additional parallax voting unit 500 performs parallax interpolation on a region where the parallax is missing from the parallax image and is difficult to detect as an object, thereby enabling the object detection unit 600 to detect an object where the parallax is missing. The internal structure of the additional parallax voting unit 500 is as follows. Figure 8 shown.

[0022] The object detection unit (object detection unit 600) identifies the outside by detecting objects based on the number of votes cast by the parallax in the UD voting space. Therefore, the object detection unit 600 can determine whether the detected object is an obstacle that the vehicle should avoid. In addition, various objects are reflected in the original image as external information, and objects that the vehicle must avoid or stop in some way are called "obstacles." Moreover, compared with previous external recognition devices, the object detection unit 600 can reliably detect obstacles even of low height. In addition, information related to obstacles detected by the object detection unit 600 (size, distance, category, etc. of the obstacle) is sent to the unillustrated automatic driving control ECU for automatic driving.

[0023] The alarm control unit 700 controls various alarms based on the information of the obstacle detected by the object detection unit 600. For example, it outputs an instruction to move the steering wheel in the direction of avoiding the obstacle or apply the brakes to the automatic driving control device (not shown), or displays an alarm message on the dashboard in the vehicle, or emits an alarm sound from the speaker in the vehicle.

[0024] <Computer Hardware Configuration Example> Next, the hardware configuration of the computer 800 constituting the external recognition device 1 will be described. Figure 2 800 is an example of hardware used as a computer that can operate as the external recognition device 1 of this embodiment. The external recognition device 1 of this embodiment and the computer 800 (computer) execute a program to realize Figure 1 The image recognition method is performed by the cooperation of the functional blocks shown.

[0025] The computer 800 includes a CPU (Central Processing Unit) 810, a ROM (Read Only Memory) 820, and a RAM (Random Access Memory) 830, which are respectively connected to a bus 840. In addition, the computer 800 includes a nonvolatile memory 850 and a network interface 860.

[0026] The CPU 810 reads the program code of the software that implements each function of the present embodiment from the ROM 820, loads it into the RAM 830, and executes it. In the RAM 830, variables or parameters generated in the middle of the operation processing of the CPU 810 are temporarily written, and these variables or parameters are read out appropriately by the CPU 810. However, an MPU (MicroProcessing Unit) may be used instead of the CPU 810, or the CPU 810 and GPU (Graphics Processing Unit) may be used together. The CPU 810 implements Figure 1 The functions of the external information acquisition unit 100 , the disparity generation unit 200 , the disparity voting unit 300 , the texture region segmentation unit 400 , the additional disparity voting unit 500 , the object detection unit 600 and the alarm control unit 700 are shown.

[0027] The nonvolatile memory 850 is, for example, a HDD (Hard Disk Drive), an SSD (Solid State Drive), a floppy disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, or a nonvolatile memory. In the nonvolatile memory 850, in addition to the OS (Operating System) and various parameters, a program for making the computer 800 function is recorded. The ROM 820 and the nonvolatile memory 850 record programs, data, etc. required for the CPU 810 to work, and are used as an example of a computer-readable non-transient storage medium that stores programs executed by the computer 800. The external information acquired by the external information acquisition unit 100, the disparity generated by the disparity generation unit 200, the number of votes for the disparity voted in the UD voting space, and the information of the object detected by the object detection unit 600 are stored in the nonvolatile memory 850 and are appropriately collected by the developer of the software program through the network.

[0028] The network interface 860 uses, for example, a NIC (Network Interface Card) and can transmit and receive various data between devices via a LAN (Local Area Network) or a dedicated line connected to a terminal of the NIC. The external recognition device 1 can transmit and receive various data from an external cloud server or the like through the network interface 860 .

[0029] <Instructions for Parallax Voting> In the right Figure 1 Before describing the internal configuration example of the texture region segmentation unit 400 and the additional parallax voting unit 500, refer to Figure 3 to Figure 6The parallax voting process and the UD voting space will be described. Here, the process of obtaining the distance frequency of each pixel of the parallax image 10 including the distance information obtained based on the parallax of the object will be described.

[0030] Figure 3 This is a diagram showing an example of the frequency distribution of distances in an area (road surface) where no three-dimensional object exists. exist Figure 3 The parallax image 10 shown on the left side of FIG. 1 shows a three-dimensional object obtained based on external information. The range of the parallax image 10 corresponds to Figure 3 The imaging range of the camera 110 (e.g., a stereo camera) shown on the right side of the figure. The horizontal axis of the parallax image 10 represents the horizontal position of the original image, and the vertical axis of the parallax image 10 represents the distance (parallax value) to the object projected in the original image calculated based on the parallax. Therefore, objects that are close are projected in front of the parallax image 10 (below), and objects that are far are projected in the back (upper). Since the parallax image 10 contains distance information, it is also called a distance image.

[0031] Assume that the road surface of the lane is reflected in the road surface area 11 on the right side of the parallax image 10. The road surface is an object with few bumps and unevenness that is evenly paved from near to far of the vehicle. Figure 3 In the description, the road surface area 11 on the right side of the parallax image 10 is limited to a range of a column 12. The column 12 is composed of Figure 3 The right side of FIG. 1 is composed of 11 blocks for calculating the distance from the camera 110. It is assumed that the voting amount of one block is "1".

[0032] exist Figure 3 The right side of the figure shows that the camera 110 is shooting the road surface area 11 within the range of the column 12. Figure 3 The camera 110 has a camera range 13 shown. Figure 3 The histogram shown in the upper right corner of Fig.14 The voting result of the distance from the position of the camera 110 to the object is represented within the range of the column 12. The parallax voting unit 300 votes the number of blocks at the distance of any block in the vertical direction of the column 12 in the road surface area 11 and the block with the same vertical position. In this way, a histogram about any vertical position is generated. The number of blocks voted by the parallax voting unit 300 is a value related to the distance. The distance is calculated based on the parallax, so in the following description, voting for blocks with equal distance for a certain distance will also be referred to as "parallax voting" or "voting for parallax".

[0033] In the straight Fig.14The threshold value of the number of votes that should be recognized as an obstacle is shown in . The threshold value is located at the position of five blocks. As described above, since the road surface has few bumps, the number of blocks that vote is about 1 to 2 blocks regardless of the distance from the camera 110, which is lower than the threshold value "5". Therefore, the road surface is not recognized as an obstacle.

[0034] Figure 4 This is a diagram showing an example of the frequency distribution of the distance in an area where a three-dimensional object (front vehicle) exists. Figure 4 The parallax image 10 shown on the left side is Figure 3 The parallax image 10 shown is the same as that shown in FIG. 1. In the vehicle region 16 on the left side of the parallax image 10, a preceding vehicle 18 traveling 10 meters ahead of the vehicle is shown (see FIG. 1 ). Figure 4 ). Figure 4 In the vehicle region 16 on the left side of the parallax image 10, the range of the column 17 is limited. The column 17 is composed of Figure 4 The right side of FIG. 1 is composed of 11 blocks for calculating the distance from the camera 110 .

[0035] exist Figure 4 The right side of the figure shows that the camera 110 is capturing the vehicle area 16 within the range of the column 17. The back of the front vehicle 18 enters the vehicle area 16 at a distance of 10m. Figure 4 The camera 110 has a camera range 13 shown. Figure 4 The histogram 19 shown in the upper right of FIG. 1 shows the voting results of the distance from the position of the camera 110 to the object within the range of the column 17. The parallax voting unit 300 votes for the number of blocks in the distance between any block in the vertical direction of the column 17 in the vehicle area 16 and the block with the same vertical position.

[0036] like Figure 4 As shown on the left side of , the block at the position of about 10m, which is the back of the front vehicle 18, accumulates very high, and 6 blocks exceeding the threshold value "5" are voted at the position of 10m in the histogram 19. Therefore, according to the histogram 19, it is determined that the back of the front vehicle 18 is located 10m in front of the vehicle.

[0037] Thus, in the area where there is no three-dimensional object and in the area where there is three-dimensional object, the histogram Fig.14 , 19. In addition, in the area where a solid object exists, the number of blocks increases according to the height of the solid object. At a distance where the number of voted blocks is greater than the threshold, the object that the vehicle should avoid exists as an obstacle.

[0038] Here, if it is Figure 4A large object such as the front vehicle 18 shown in the figure is easy to detect as a three-dimensional object, but a boundary such as a road end without a step having almost no height is difficult to identify because the accumulated number of blocks is small and the number of blocks is less than the threshold. Therefore, in the automatic driving control, it is necessary to be able to detect objects that are difficult to identify only by voting for blocks at the same distance from the camera 110 (for example, a road end without a step, a road end with a low step height, a traffic cone lying on the lane, etc.).

[0039] in addition, Figure 3 and Figure 4 The result of the parallax voting according to the distance of the object, that is, the number of votes for the parallax corresponding to the distance of the object, is reflected in the UD voting space described later. In addition, as shown in the columns 12 and 17 of the parallax image, the UD voting space is a space in which the number of votes for the parallax of the entire parallax image is reflected by staggering the columns, with the horizontal axis being the horizontal position of the original image on the screen and the vertical axis being the parallax value (distance) to the object.

[0040] Figure 5 is a diagram showing an example of a parallax image 10 and a UD voting space 20 . exist Figure 5 The upper side indicates Figure 3 and Figure 4 The parallax image 10 shown. In the parallax image 10, a road surface area 11 where a road surface exists and a vehicle area 16 where a preceding vehicle exists are shown.

[0041] exist Figure 5 The UD voting space 20 is shown below, which shows the spatial characteristics. The horizontal axis of the UD voting space 20 represents the horizontal position U corresponding to the parallax image 10, and the vertical axis represents the distance to the three-dimensional object (parallax value D). The UD voting space 20 is obtained by Figure 3 and Figure 4 The method shown in the figure visualizes the number (frequency) of blocks that vote by distance for the entire parallax image 10. The frequency of the number of blocks shown as an example in the UD voting space 20 is Figure 3 and Figure 4 the histogram Fig.14 , 19, and is obtained within the range of "0" to "7". The darker part of the UD voting space 20 indicates a lower frequency, and the lighter part indicates a higher frequency. For example, the UD voting space 20 for a part with a frequency of "0" to "1" is displayed darker (black), and the UD voting space 20 for a part with a frequency of "6" to "7" is displayed lighter (white).

[0042] Then, by changing the vertical position of the parallax image 10, a process of coloring is performed according to the frequency of the blocks that voted for a certain vertical position (distance) of the parallax image 10, thereby generating the UD voting space 20. Figure 5In the UD voting space 20 shown, the lower right area 21 indicates the presence of a step-free road end, and the upper left area 22 indicates the presence of the front vehicle 18. In this way, by using the UD voting space 20 colored according to the frequency of the blocks, objects with height are easy to detect, but objects with low height are still difficult to detect. For example, in the step-free road end shown in area 21 of the UD voting space 20, the height of the boundary of the sidewalk is slightly higher than that of the lane, so the frequency of the block is higher than that of the road surface. However, the height of the boundary is low compared to the height of the back of the front vehicle 18, so the frequency of the blocks according to the distance is generally low.

[0043] Here, the parallax voting performed by the external recognition device 1 according to the present embodiment in order to better detect a three-dimensional object will be described. Figure 6 3 is a diagram showing an example of an original image 30 captured by a stereo camera and a UD voting space 35 . Figure 6 The original image 30 shown is an image captured by the left camera of the stereo camera.

[0044] exist Figure 6 The upper part of the figure shows an original image 30 showing two front vehicles 31 and 32. The front vehicle 31 is traveling in the same lane as the vehicle, and the front vehicle 32 is on the way from an adjacent lane into the lane in which the vehicle is traveling. In addition, a central separation strip 33 is provided on the right side of the lane in which the vehicle is traveling.

[0045] exist Figure 6 An example of a UD voting space 35 generated based on the original image 30 is shown below the original image 30. The UD voting space 35 is generated based on the parallax generated from the two original images 30. The horizontal axis of the UD voting space 35 is the lateral position U corresponding to the lateral coordinate of the original image 30, and the vertical axis is the parallax value D representing the distance to the object detected from the original image 30. Objects in the UD voting space 35 are located by UD coordinates. In this way, the UD voting space has an external depth distance on one axis and an angle at which the external information acquisition unit (external information acquisition unit 100) can acquire external information, or a lateral position of the image when the external information is an image, on the other axis. The angle at which external information can be acquired is, for example, equivalent to the photographic field of view in the horizontal direction of a stereo camera, or the measurement angle in the horizontal direction that can be measured by LiDAR.

[0046] A large parallax value D indicates that the depth (distance) of the object reflected in the original image 30 is close to the stereo camera (the vehicle). Conversely, a small parallax value indicates that the depth (distance) of the object reflected in the original image 30 is far from the stereo camera (the vehicle). Figure 5 As shown, in the UD voting space 35, the voting results of the point group of parallax generated according to the distance to the object are shown.

[0047] In the UD voting space 35, the parallax is voted for at a position where the entire back side of the preceding vehicle 31 is at the same distance (for example, 20m) from the vehicle. Therefore, the parallax that votes for the preceding vehicle 31 at a position of 20m of the parallax value D is accumulated, and a white horizontal line is displayed in accordance with the lateral width of the preceding vehicle 31. On the other hand, the preceding vehicle 32 is projected obliquely in the original image 30. Therefore, the distance to the back side and the right side of the preceding vehicle 32 is detected. As a result, in the UD voting space 35, the parallax that votes for the back side of the preceding vehicle 32 at a position of 10m of the parallax value D is accumulated. In addition, the parallax that votes for the right side of the preceding vehicle 32 in the depth direction is accumulated in an amount corresponding to the vehicle length of the preceding vehicle 32. The back side and the right side of the preceding vehicle 32 are indicated by horizontal white lines and oblique white lines in the UD voting space 35. As shown Figure 6 As shown, since the height H2 of the preceding vehicle 32 closer to the host vehicle is greater in the longitudinal direction than the height H1 of the preceding vehicle 31 farther from the host vehicle, the white line representing the preceding vehicle 32 is lighter (whiter) than the white line representing the preceding vehicle 31 .

[0048] exist Figure 6 The lower part of shows an example of voting results. The voting results are simple graphs generated corresponding to the UD voting space 35. As shown in the voting results, at the position of 10m, the number of votes for the parallax of the front vehicle 32 is "7", and at the position of 20m, the number of votes for the parallax of the front vehicle 31 is "6". Since the height H2 of the front vehicle 32 in the original image 30 is higher than the height H1 of the front vehicle 31, the number of votes for the front vehicle 32 becomes larger.

[0049] In addition, since the height H3 of the central separation band 33 shown on the right side of the original image 30 is low, the number of votes for the parallax at the position corresponding to the central separation band 33 in the UD voting space 35 is "1" or "2". In addition, the number of votes in the UD voting space 35 is also recorded in other units, but it is omitted because the drawing becomes complicated. Figure 3 and Figure 4 As shown, the threshold value for identifying as an obstacle is “5.” Therefore, since the center separation strip 33 is not identified as an obstacle, a method for increasing the number of votes for the center separation strip 33 is required.

[0050] <Example of Internal Structure of Texture Region Segmentation Unit> Next, an example of the internal structure of the texture region dividing unit 400 will be described. Figure 7 : is a block diagram showing an example of the internal structure of the texture region segmentation unit 400 . The texture region segmentation unit (texture region segmentation unit 400) includes: a local similar texture region segmentation unit (local similar texture region segmentation unit 410), which divides external information into texture regions of each locally similar texture; a semantic region segmentation unit (semantic region segmentation unit 420), which divides external information into semantic regions of each object category; and an individual object region segmentation unit (individual object region segmentation unit 430), which divides external information into individual object regions for each object.

[0051] The local similar texture region segmentation unit 410 performs segmentation according to local similar textures in the captured original image. Similar textures must be at least part of the same object. For example, in the case where there is a front vehicle, if there is a similar texture (e.g., the same color) in the front vehicle, segmentation is performed according to the similar texture.

[0052] The semantic region segmentation unit 420 segments various objects reflected in the captured original image into semantic regions for each object, such as vehicles and roads. Such regions segmented according to the semantics of each object are called "semantic regions". As semantic region segmentation, for example, a segmentation method called semantic segmentation is used. Semantic segmentation is a method of segmenting regions in an image according to the category of an object by assigning specific labels to all pixels in the image. The regions segmented by semantic segmentation are represented, for example, as described below. Fig. 9 The semantic region segmented image 45 is obtained. In the semantic segmentation, when a plurality of vehicles are overlapped and projected on the original image, the plurality of vehicles are segmented from the original image as a whole as a semantic region. That is, in the semantic region, a plurality of vehicles overlapped and projected in the original image are not segmented as individual vehicles. The specific example of the processing of the semantic region segmentation unit 420 is described through the semantic region segmentation processing described later.

[0053] The individual object region segmentation unit 430 segments the captured original image into regions according to individual objects such as vehicles and roads. For example, a segmentation method called instance segmentation is used as the individual object region segmentation. Instance segmentation is a method of detecting overlapping objects in an image and segmenting the region in the image into each detected object. The regions segmented by instance segmentation are described later. Fig.13 The region including the object individually segmented from the image is referred to as a “individual object region”. In the individual object region segmentation process described later, a specific example of the process performed by the individual object region segmentation unit 430 will be described.

[0054] In the following description, when texture areas, semantic areas, and individual object areas are not distinguished, they are sometimes collectively referred to as “segmented areas”.

[0055] <Example of the internal structure of the additional parallax voting unit> Next, an example of the internal structure of the additional parallax voting unit 500 will be described. Figure 8 2 is a block diagram showing an example of the internal configuration of the additional parallax voting unit 500 . The additional parallax voting unit 500 includes a non-step road edge additional voting unit 510 , a road surface paint subtraction unit 520 , a class obstacle interpolation voting unit 530 , a single obstacle interpolation voting unit 540 , and a local area interpolation voting unit 550 .

[0056] The step-free road end additional voting unit 510 performs additional disparity voting in the UD voting space for road ends without steps or with low steps (referred to as "step-free road ends"). At step-free road ends, since the number of votes for disparity is small, the number of votes for the step-free road end is lower than the threshold. Therefore, the step-free road end additional voting unit 510 adds disparity to the step-free road end, thereby increasing the number of votes for the disparity of the step-free road end. Through such processing, the object detection unit 600 can detect the step-free road end as an obstacle. The details of the processing of the step-free road end additional voting unit 510 will be described later. Fig. 9 and Fig.10 Provide explanation.

[0057] The additional parallax voting unit (additional parallax voting unit 500) has a subtraction unit (road paint subtraction unit 520) that reduces the number of votes for the parallax that votes for the road surface markings. For example, as an example of a specific texture area, a road surface marking painted on the road surface (called "road paint") such as a zebra crossing area, a crosswalk, and a stop sign, when painted in the depth direction, although there is no height as an object, parallax sometimes occurs at the boundary between the road surface and the white line, and it is easy to be misjudged as an obstacle. In addition, if the white line is drawn to cross the lane, no parallax will be generated, and the distance to the white line becomes unclear. However, through the processing of the semantic area segmentation unit 420 of the present embodiment, it is determined that the part is painted on the road surface. Therefore, the road paint subtraction unit 520 performs a weighted subtraction of the number of votes for the parallax that the parallax voting unit 300 voted for the road surface paint. When the region labeled with road paint is segmented by the texture region segmentation unit 400 , the road paint subtraction unit 520 adjusts the number of votes for the parallax voted by the parallax voting unit 300 for the region labeled with road paint.

[0058] For example, the road paint subtraction unit 520 subtracts half of the voting value of the parallax voted by the parallax voting unit 300 from the voting value of the road paint. Here, it is assumed that the unit amount of the number of votes voted by the parallax voting unit 300 in the UD voting space is usually "1". The road paint subtraction unit 520 performs weighting by subtracting the number of votes of the parallax that has already voted by a unit amount such as "0.5". In addition, the road paint subtraction unit 520 can also cancel the number of votes of the parallax voting unit 300 for the segmented area with the road paint label by weighting the number of votes of the parallax that has already voted by voting "-1". In this way, in order to prevent the segmented area with the road paint label from being mistakenly detected as an obstacle by the object detection unit 600, the road paint subtraction unit 520 adjusts the segmented area so that a large number of votes will not be gathered. As a result, when additional voting is performed in the additional parallax voting unit 500, the number of votes for the area with the road paint label is adjusted.

[0059] However, the road paint subtraction unit 520 may also adjust the number of votes (height) of a specific part in the UD voting space after the additional voting is completed. For example, assume that the position of the road paint in the UD voting space at U=100 and D=10 is determined based on the parallax value. At this time, the road paint subtraction unit 520 halves the number of votes (height) of the voting space near the road paint area. In addition, in order to prevent the voting value of the parallax voting for the road paint from changing extremely from the voting value of the surrounding part of the painted part, a smoothing filter may be applied near the road paint area to reduce the number of votes (height) near the area. In addition, the road paint subtraction unit 520 may also subtract the number of votes from the road marking projected on the unpainted road surface. In this way, by assuming that the road paint is a plane, the voting value of the painted part is dispersed to the voting value of the surrounding part of the painted part, which can reduce the possibility of misjudging the painted part as an obstacle. In addition, the object detection unit 600 will no longer misdetect the road paint as an obstacle.

[0060] The category obstacle interpolation voting unit 530 votes for the category of the interpolated obstacle. For example, when multiple objects are reflected in the original image, if they are objects of the same category, the category obstacle interpolation voting unit 530 interpolates the category of the multiple objects as a whole regardless of the positions of the multiple objects. Objects are classified into categories such as vehicles, pedestrians, roads, white lines, and road paint. Objects such as vehicles and pedestrians are required to be reliably detected as obstacles, so parallax is interpolated according to the category of these objects to suppress the non-detection of these objects. Therefore, the category obstacle interpolation voting unit 530 determines that in Figure 3 and Figure 4The object detection unit 600 detects a group of objects with heights such as pedestrians or vehicles as obstacles, and detects whether the same distance is assumed in the vertical column of the image. If the same distance is assumed in the vertical column, the detected object has a certain height at the distance. On the other hand, if the same distance is not assumed in the vertical column, the height of the detected object is low. By such processing, the object detection unit 600 will no longer mistakenly detect road paint such as white lines as obstacles, and can detect a group of objects with heights such as pedestrians or vehicles as obstacles.

[0061] The individual obstacle interpolation voting unit 540 performs voting for individual categories of obstacle interpolation. For example, when multiple objects are reflected in the captured original image, the individual obstacle interpolation voting unit 540 interpolates parallax for each object individually. For example, different identification information (identification ID) is assigned to detected three-dimensional objects such as vehicles or pedestrians by the individual object region segmentation unit 430. When the individual obstacle interpolation voting unit 540 detects multiple pedestrians crossing the crosswalk, these pedestrians can be grouped together as pedestrians at the same distance to interpolate parallax. In addition, for the vehicle in front, as long as it is within the same object, parallax can be interpolated as a vehicle at the same distance. Through such processing, the object detection unit 600 can detect objects with height as obstacles individually, or detect multiple objects as obstacles collectively.

[0062] The local area interpolation voting unit 550 performs voting for interpolating the local similar texture area segmented by the local similar texture area segmentation unit 410. Here, the local similar texture area is also referred to as a "local area". In the original image, the same texture in the same object is segmented into local areas. Then, if it is within the local area similar texture, the local area interpolation voting unit 550 assumes that all are the same object, the same distance, or the same plane, and votes for the parallax. On the other hand, if the parallax dispersion is large, the local area interpolation voting unit 550 sometimes does not interpolate the parallax even within the local area similar texture area. For example, if the parallax is dispersed at positions of 5m and 20m, it is considered that the textures of different objects happen to be the same, so the parallax is not interpolated. Through such processing, the object detection unit 600 can detect objects of the same texture dispersed in the same object as obstacles to one object.

[0063] <Description of semantic region segmentation> Next, refer to Fig. 9 and Fig.10 Illustrate semantic region segmentation. Fig. 9 FIG. 1 is a diagram showing transitions of images for explaining semantic area segmentation. Fig.10 FIG. 1 is a diagram showing examples of voting results in each image.

[0064] exist Fig. 9 On the upper side of the horizontal line 41, one of the two images taken by the stereo camera (for example, the right image taken by the right camera) is represented as the original image 40. The original image 40 is represented by UV coordinates with the horizontal axis as the horizontal position U and the vertical axis as the vertical position V. The horizontal line 41 that crosses the original image 40 is set to distinguish the objects of image processing and is not reflected on the actual original image 40. Since the area below the horizontal line 41 contains the information required for vehicle driving, obstacle detection is performed on the area below the horizontal line 41. The area above the horizontal line 41 is the upper side of distant buildings, roads, trees, etc. Since only objects that the vehicle has no possibility of contacting are reflected, obstacle detection is not required in this area.

[0065] In addition, a processing area 42 surrounded by white lines is shown in the original image 40. Assuming that the vehicle is traveling in a straight line in left-hand traffic, the vehicle needs to maintain driving in the same lane except for the case of lane changes. Therefore, an area including the boundary between the sidewalk and the lane on the left side of the lane in which the vehicle is traveling and an area including the boundary between the sidewalk and the lane of the opposite lane are selected as the processing area 42, and parallax voting, parallax addition, etc. are performed on the processing area 42. In addition, the entire area below the horizontal line 41 can also be selected as the processing area 42. Processing numbers (1) to (4) are attached to the arrows shown in the figure to explain the content of each processing.

[0066] (1) Processing of generating parallax images The parallax generation unit 200 generates a parallax generated from two images captured by a stereo camera. The parallax generated by the parallax generation unit 200 is illustrated as a parallax image 43 in the figure. The parallax image 43 is represented by the same UV coordinates as the original image 40. In the parallax image 43, an area with a large parallax is displayed on the near side, and an area with a small parallax is displayed on the far side. Since objects with height such as the preceding vehicle, a person, a traffic light, etc. are not reflected in the original image 40, the parallax of an object that can become an obstacle to the vehicle is not included in the parallax image 43.

[0067] (2) Voting process in UD voting space The parallax voting unit 300 performs voting processing based on the distance of the detected object according to the parallax image 43 to generate a UD voting space 50. The UD voting space 50 is represented by a UD coordinate with U as the horizontal axis and D (depth (parallax)) as the vertical axis. As described above, the parallax image 43 does not include the parallax of objects that can become obstacles to the vehicle, and since the step at the boundary between the lane and the sidewalk is small, the boundary 51 is blurred and unclear. If the parallax image 43 is used for autonomous driving control, the position of the vehicle within the lane may become unclear. Therefore, Figure 7The semantic area segmentation unit 420 shown performs semantic area segmentation on the original image 40 .

[0068] exist Fig.10 The upper side of represents the number of votes for the parallax voted in the UD voting space 50. Here, attention is paid to the number of votes for the parallax voted on the boundary between the lane and the sidewalk whose edge is emphasized in the parallax image 43 in the UD voting space 50. Fig.10 In the voting results shown on the upper side of , there are only votes ("2" to "4") with values ​​smaller than the threshold "5" at the boundary between the lane and the sidewalk, so Figure 1 The object detection unit 600 shown cannot correctly identify the boundary.

[0069] (3) Semantic region segmentation The semantic region segmentation unit 420 performs semantic region segmentation on each pixel of the original image 40. When performing semantic region segmentation, the original image 40 is represented as a semantic region segmentation image 45 in which the region is segmented according to the semantics of the object. The semantic region segmentation image 45 is an image in which the original image 40 is segmented into a region 46 representing the road surface, a region 47 representing the white line that determines the lane, and a region 48 including the sidewalk outside the lane (see Fig. 9 ). However, the processing area 42 shown in the original image 40 includes the boundary between the area 46 showing the road surface and the area 48 including the sidewalk other than the lane, so it is necessary to detect the boundary between the areas 46 and 48.

[0070] exist Fig.10 The middle of shows the disparity (number of votes) added based on the semantic region segmentation image 45. In the semantic region segmentation image 45, the boundary 49 between the lane and the sidewalk is clearly shown. Therefore, the disparity is added to the boundary 49. The added disparity can be the same value or different values. However, it is necessary to add the disparity in such a way that the number of votes for the boundary that was originally unclear in the UD voting space 50 exceeds the threshold.

[0071] (4) Voting Processing from Semantic Region Segmentation Image to UD Voting Space The stepless roadside additional voting unit 510 of the additional parallax voting unit 500 performs voting processing according to the distance of the detected object to generate a UD voting space 55. At this time, the stepless roadside additional voting unit 510 performs additional voting on the boundary between the lane and the sidewalk as the stepless roadside based on the semantic area segmentation image 45 generated by the semantic area segmentation unit 420. The UD voting space 55 is represented by UD coordinates. In the UD voting space 55, by additionally voting parallax based on the semantic area segmentation image 45, the boundary 56 between the lane and the sidewalk is more clearly represented (white emphasis) compared to the UD voting space 50 generated only from the parallax image 43. The boundary 56 represents the characteristics of the stepless shoulder. Therefore, the vehicle can recognize the lane on the inner side of the stepless shoulder and travel.

[0072] exist Fig.10 The lower side of represents the voting result of adding parallax in UD voting space 50. In this voting result, the number of votes for the part with added parallax is greater than the threshold value "5". Therefore, Figure 1 The object detection unit 600 shown can correctly detect the boundaries of lanes and sidewalks. Fig. 9 and Fig.10 The processing can also be performed on the texture area segmented by the local similar texture area segmentation unit 410.

[0073] Next, refer to Fig.11 and Fig.12 The process of performing voting after parallax interpolation of obstacles (referred to as "obstacle interpolation voting") will be described. Fig.11 is a diagram showing an example of obstacle interpolation voting after semantic region segmentation. Fig.11 In the example, the original image is omitted. Fig.12 FIG. 1 is a diagram showing examples of voting results in each image.

[0074] exist Fig.11 An example of a parallax image 60 is shown in the upper left corner of FIG. In the parallax image 60, a vehicle traveling in the same direction of travel as the vehicle itself and a plurality of vehicles traveling in the opposite lane are shown. Here, if the back of the vehicle traveling in the same direction of travel as the vehicle itself is black, or if it is traveling at night, the parallax of the vehicle traveling in the same direction of travel as the vehicle itself is sometimes missing in part of the area 61. Similarly, the parallax of the vehicle traveling in the opposite lane is sometimes missing in part of the area 61. This lack of parallax in a portion that originally had parallax is also called "parallax loss". Parallax loss tends to occur in portions of the original image where the texture is weaker, such as portions where there is no difference in light and dark.

[0075] (1) Voting Process from Parallax Image 60 to UD Voting Space When voting is performed from the parallax image 60 to the UD voting space 65, the number of votes in the parallax-missing region 61 is less than the threshold value as shown in region 66. If the number of votes is less than the threshold value, the object detection unit 600 may mistakenly detect one vehicle as multiple different objects.

[0076] exist Fig.12 The upper side of represents the number of votes for the parallax voted in the UD voting space 65. Here, attention is paid to the boundary of the lane and the sidewalk whose edges are emphasized in the parallax image 60 in the UD voting space 65, and the number of votes for the parallax voted on the surface of the vehicle facing the own vehicle. Fig.12 In the voting results shown on the upper side of , since there is a vote number "6" with a value greater than the threshold "5" at the boundary between the lane and the sidewalk, Figure 1 The object detection unit 600 shown can correctly identify the boundary. On the other hand, since the surface of the vehicle facing the vehicle lacks parallax in the area 61, there is only a vote number "2" which is a value smaller than the threshold value "5". Therefore, the object detection unit 600 cannot correctly identify the size, width, and direction of travel of the vehicle.

[0077] Therefore, Figure 7 The semantic region segmentation unit 420 shown in the figure studies the semantic region segmentation image 70 in which the semantic region segmentation is performed for each pixel in the original image. Fig.11 An example of a semantic area segmentation image 70 is shown in the upper right corner of FIG. The semantic region segmentation image 70 shows the regions in which the original image is semantically segmented. The semantic region segmentation image 70 divides the original image into a region 71 representing the road surface, a region 72 representing the road surface and other than the vehicle, and a region 73 representing the vehicle. In addition, in the semantic region segmentation, the four vehicles projected on the original image are collectively represented by a region 73 of the vehicle category. In addition, each region 71 to 73 is distinguished by painting a different color on each object category, that is, the regions 71 to 73. In addition, when the regions 71 to 73 painted with colors are not distinguished, they are called "painted regions."

[0078] Next, the process of acquiring the parallax image 60 , the process of performing parallax interpolation on the parallax image 60 , and the process of performing voting in the UD voting space based on the interpolated parallax image 60 are described separately.

[0079] (2) Parallax Image Acquisition and Processing First, a case where a parallax image 60 with parallax interpolated is obtained will be described. The obstacle interpolation voting unit 530 (see Figure 8 ) from the parallax generating unit 200 (refer to Figure 1) obtains the parallax image 60 and interpolates the parallax of the missing area 61 of the parallax image 60.

[0080] (3) Interpolation of parallax images Next, the obstacle interpolation voting unit 530 interpolates the disparity of the obstacle detected from the disparity image 60 by category. Here, the obstacle interpolation voting unit 530 assumes that the obstacle detected in the disparity image 60 is a plane with the same depth in the height direction (vertical direction in the image), and interpolates the part where the disparity is missing with the disparity around the obstacle.

[0081] As a method for the obstacle-class interpolation voting unit 530 to interpolate parallax, the following methods (a) to (c) are conceivable. (a) Method of interpolating parallax using the vertical mode value in the parallax image 60 The class obstacle interpolation voting unit 530 of the additional disparity voting unit (additional disparity voting unit 500) interpolates the disparity missing part existing in the part corresponding to the texture area of ​​the disparity image having a depth distance on one axis with the disparity existing in the axial direction of the depth distance of the disparity image. Then, the class obstacle interpolation voting unit 530 votes for the additional disparity according to the texture area in which the disparity missing part is interpolated. Here, the class obstacle interpolation voting unit 530 selects the interpolated disparity according to the attribute of the semantic area. For example, for the disparity missing part of the semantic area whose attribute is an object such as a vehicle or a person, the class obstacle interpolation voting unit 530 interpolates using the disparity existing in the axial direction of the depth distance of the disparity image, that is, in the longitudinal direction. On the other hand, for the disparity missing part of the semantic area whose attribute is a planar object such as a road surface, the class obstacle interpolation voting unit 530 interpolates using the disparity in the other axial direction of the depth distance of the disparity image, that is, in the lateral position. By performing interpolation with the parallax matching the type of the object in this way, the parallax after interpolation is less likely to fluctuate than the surrounding parallax.

[0082] (b) A method of interpolating the parallax missing part using the pixel values ​​of eight surrounding pixels tilted up, down, left, and right around the parallax missing part in the coloring area. The obstacle interpolation voting unit 530 of the additional parallax voting unit (the additional parallax voting unit 500) interpolates the parallax missing part existing in the portion corresponding to the texture area of ​​the parallax image having a depth distance on one axis using the parallax around the parallax missing part existing in the same object as the object where the parallax missing part exists. Then, the obstacle interpolation voting unit 530 votes for the additional parallax according to the texture area where the parallax missing part is interpolated. For example, the parallax missing part is interpolated using the pixel values ​​of eight surrounding pixels tilted up, down, left, and right around the parallax missing part, so that the parallax missing part does not have an extremely different value from the surrounding pixels.

[0083] (c) Method of interpolating using the parallax of the areas above and below the parallax missing area For example, the edges detected at the boundaries between the background and the object, the road surface and the object are prone to parallax, but since the edges themselves have no thickness, the parallax is easily interrupted. In addition, the portion with texture can determine the same part of the left and right images, so it is easy to produce parallax, while the portion without texture is difficult to determine the same part of the left and right images, so it is difficult to produce parallax. Therefore, for the parallax interrupted or parallax-deficient area where parallax is difficult to produce, the category obstacle interpolation voting unit 530 performs interpolation processing using the parallax of the area containing parallax near the area. Therefore, the category obstacle interpolation voting unit 530 can obtain continuous parallax.

[0084] In addition, in the above methods (b) and (c), it is possible to interpolate the parallax missing part using the black color (low frequency of parallax) which is the most in the vertical direction of the parallax image. Therefore, the obstacle classification interpolation voting unit 530 only treats black as an invalid parallax instead of a valid parallax value. Then, the invalid parallax part (black area) lacking parallax is interpolated using the parallax of the surrounding area (part other than black) of the invalid parallax part.

[0085] In addition, the category obstacle interpolation voting unit 530 may interpolate the semantic area using the parallax weighted in proportion to the area of ​​the semantic area. For example, when there are multiple leading vehicles, the weight of the parallax relative to the leading vehicle that is close to the vehicle and has a large semantic area is made greater than the weight of the parallax relative to the leading vehicle that is far from the vehicle and has a small semantic area. In this way, by weighting in proportion to the area of ​​the semantic area, the object detection unit 600 can reliably detect the leading vehicle approaching the vehicle.

[0086] The obstacle classification interpolation voting unit 530 interpolates the parallax at the parallax missing portion and displays the image as the parallax interpolation image 75. Fig.11 The area 78 of the parallax interpolation image 75 is interpolated. Fig.11 The parallax missing portion shown in the upper left of FIG. 6 is the parallax portion of the region 61. The parallax interpolation or other processing is not performed on the portion other than the region 61 of the parallax image 60.

[0087] exist Fig.12 The middle part shows the image 70 segmented based on semantic regions (see Fig.11 ) is added to the parallax (number of votes) of the parallax interpolation image 75. In the parallax interpolation image 75, the parallax is interpolated to the area 78 (see Fig.11). The interpolated disparity may be the same value or different values. However, the disparity needs to be interpolated in such a way that the number of votes of the disparity area 66 originally missing in the UD voting space 65 exceeds a threshold.

[0088] (4) Voting process in UD voting space based on parallax interpolation image The class obstacle interpolation voting unit 530 of the additional parallax voting unit 500 performs a voting process according to the class of the object detected from the parallax interpolation image 75, and generates a UD voting space 80. The UD voting space 80 is represented by UD coordinates. The class obstacle interpolation voting unit 530 interpolates the parallax added by the class obstacle interpolation voting unit 530 to the parallax missing part in the UD voting space 65 that originally lacks parallax.

[0089] exist Fig.12 The lower side of represents the voting result after the additional parallax is interpolated to the UD voting space 80. In this voting result, the number of votes for the parts after the additional parallax is interpolated is a value "6" which is larger than the threshold value "5". Therefore, Figure 1 The object detection unit 600 shown is capable of Fig.11 The width, size, and presence of the vehicle are correctly recognized in the illustrated area 81. As a result, the vehicle can travel while recognizing the lane.

[0090] In the conventional process of directly voting from the parallax image 60 to the UD voting space, if the parallax of the object is missing in the area 66 which is the parallax missing part, the possibility of not detecting the obstacle in the conventional object detection unit is high. On the other hand, compared with the processing result of voting the parallax from the parallax image 60 to the UD voting space, the object detection unit 600 can detect the object stably as a set by adding the parallax to the parallax missing part by the obstacle class interpolation voting unit 530. Therefore, the possibility of the object detection unit 600 detecting the obstacle can be increased.

[0091] Next, a description will be given of a process of adding parallax to a parallax-missing portion of an individual object region. Fig.13 is a diagram showing an example of obstacle interpolation voting after performing individual object region segmentation. Fig.13 In the example above, the original image is omitted. Fig.11 and Fig.13 In the present invention, since substantially the same images are used, detailed processing instructions are omitted.

[0092] (1) Voting Process from Parallax Image 60 to UD Voting Space When the voting process from the parallax image 60 to the UD voting space 65 is performed, the number of votes for the parallax missing region 61 becomes smaller than the threshold value as shown in the region 66. Fig. 9 As described above, the object detection unit 600 may mistakenly detect one vehicle as a plurality of different objects.

[0093] Therefore, Figure 8 The following describes a process in which the individual object region segmentation unit 430 shown in FIG. 1 generates an individual object region segmentation image 70A in which the individual object region is segmented for each pixel based on the original image. Fig.13 An example of a single object region segmented image 70A is shown in the upper right corner of FIG.

[0094] The separate object region segmentation image 70A shows the separate object regions obtained by segmenting the original image into separate object regions. The separate object region segmentation image 70A is an image of separate object regions in which the original image is segmented into a region 71 representing the road surface, a region 72 representing the road surface and other than the vehicle, and regions 73a to 73d representing the vehicle. In the separate object region segmentation, the four vehicles shown in the original image are represented by separate regions 73a to 73d for each vehicle. Furthermore, the separate object region segmentation unit 430 labels the separate regions 73a to 73d of each vehicle with "car1" to "car4".

[0095] (2) Interpolation of parallax images first, Figure 8 The individual obstacle interpolation voting unit 540 shown interpolates the parallax individually for the object having the parallax missing part among the objects detected in the parallax image 60. Here, the individual obstacle interpolation voting unit 540 interpolates the parallax in the same object among the obstacles detected in the parallax image 60. The method of interpolating the parallax may be a method of interpolating from the area surrounding the area lacking the parallax, or a method of interpolating the parallax in the area containing the parallax above and below the area lacking the parallax. In addition, if the posture of the object is not considered, the distance to each part of the same object may also be considered to be the same. For example, although the front and side of the vehicle traveling in the opposite lane are at different distances from the vehicle, the distance to the side of the vehicle can be considered to be the same as the distance to the front. Therefore, even if the obstacle is located at a long distance, the distance of the object can be individually calculated by the individual object region segmentation unit 430, so the parallax of the parallax missing part of the object can be individually interpolated.

[0096] (3) Processing of interpolating the additional parallax of the parallax missing portion of the parallax image 60, (4) Processing of voting the parallax added to the parallax interpolation image 75 in the UD voting space, as shown in FIG. Fig.11 As described.

[0097] In this way, the individual object region segmentation unit 430 identifies the objects reflected in the original image individually and segments them into individual object regions, thereby being able to correctly identify each object. Therefore, compared with the process of directly voting from the parallax image 60 to the UD voting space, the possibility of detecting each object individually can be increased.

[0098] Here, several obstacle detection methods are described. In the object detection method based on the above-mentioned semantic region segmentation, the texture region segmentation unit 400 performs region segmentation while connecting the pixel values ​​of 8 pixels near the focus pixel of the UD voting space (8 pixels centered on the focus pixel, up, down, left, right, top right, bottom right, top left, and bottom left). However, it is also possible to perform region segmentation while connecting the pixel values ​​of 8 pixels near the focus pixel of the UD voting space (pixels centered on the focus pixel, up, down, left, and right). In addition, the texture region segmentation unit 400 assigns the same ID to the part of the vote for a three-dimensional object such as a car, so the object detection unit 600 can detect the part as an object. In addition, by adding parallax to the part where parallax is missing, the object detection unit 600 can analyze the three-dimensional vehicle shape that is difficult to perform only by segmenting the object region alone, and can also analyze the lateral position, posture, and shape of the vehicle. Here, two examples of processing to suppress the object detection unit 600 from erroneously detecting objects that may become obstacles are described below.

[0099] <False detection suppression processing 1> First, the case where the separate object region segmentation unit 430 performs an erroneous separate object region segmentation process is described. For example, it is assumed that the separate object region segmentation unit 430 obtains an erroneous separate object region segmentation result indicating that there are objects such as road ends or vehicles on the road surface that may become obstacles. As described above, the separate obstacle interpolation voting unit 540 votes for the parallax after the road surface is interpolated in the UD voting space. Here, in the UD voting space, the votes are dispersed in the depth direction on the road surface, and a large number of votes cannot be obtained in the separate object region that is erroneously divided into an object existing on the road surface. Therefore, even if the separate object region segmentation unit 430 erroneously divides the separate object region, the erroneous detection result can be suppressed. As a result, the object on the road surface that the vehicle should avoid is correctly detected by the object detection unit 600.

[0100] <False detection suppression process 2> In addition, the case where the semantic region segmentation unit 420 performs an erroneous semantic region segmentation process is described. Here, it is assumed that the result of the semantic region segmentation is, for example, a result that not only an object that is an original obstacle but also an area of ​​the road surface that includes the background of the object is captured as a large obstacle. In this case, in the semantic region segmentation, the original image is segmented into an area that is much larger than the original obstacle. However, Figure 8 The additional parallax voting unit 500 shown confirms the voting result of the parallax of the divided area and adds the correct parallax to the obstacle. Therefore, the object detection unit 600 can accurately detect the original width, shape, and posture of the obstacle.

[0101] Unlike the conventional method of identifying based on the result of a single region segmentation, the false detection suppression processes 1 and 2 are processes for obtaining voting results by interpolating the parallax. Furthermore, through the false detection suppression processes 1 and 2, the object detection unit 600 can correctly detect the three-dimensional shape of the object and suppress false detection of obstacles.

[0102] <Processing of Local Area Interpolation> Next, the processing of local area interpolation will be described. Fig.14 is a diagram for explaining the process of recognizing traffic cones. Fig.14 The upper left of shows an example of the original image 90 showing a traffic cone placed on the road. Fig.14 The lower left of is an example of a parallax image 95 generated from the original image 90 .

[0103] exist Fig.14 The right side of shows a three-dimensional point group and an estimated plane. As described above, the parallax image is composed of a three-dimensional point group. In the previous external recognition device, objects with curved surfaces such as conical traffic cones are sometimes mixed with the background, making it difficult to recognize only the object. Therefore, Figure 7 The local similar texture region segmentation unit 410 shown in FIG. 4 approximates the point group on the same unit acquired by LiDAR, for example, to a plane. Fig.14 The shape of an object such as the traffic cone shown on the lower left is called a unit.

[0104] Figure 8The local area interpolation voting unit 550 shown determines whether the plane of the local area segmented by the local similar texture area segmentation unit 410 is the side of the same three-dimensional object based on the error amount between the plane of the local area segmented by the local similar texture area segmentation unit 410 and the point group acquired by LiDAR. For example, in the local area identified by the local similar texture area segmentation unit 410 by mixing the background and the object, the error amount of the point group of the background and the object that are not the same object is large. On the other hand, compared with the error amount of the point group of the object that is not the same object (for example, the side of a traffic cone) and the background, the error amount of the plane of the local area and the point group of the object that is the same object (for example, the side of a traffic cone) is small, even if the object has a curved surface.

[0105] Therefore, if Fig.14 As shown on the right side of , the local area interpolation voting unit 550 confirms whether the point group of the local area is in the same depth or the same three-dimensional plane. In addition, when it can be determined that a unit is an object, the local area interpolation voting unit 550 performs depth interpolation in the unit. After that, the local area interpolation voting unit 550 can interpolate invalid parallax parts or noise by using the unit interpolated for depth to vote for additional parallax.

[0106] However, even when the local area interpolation voting unit 550 is able to approximate the point group with a plane of the local area, if the approximated plane is close to the position and inclination of the road surface, the object (plane) determined in the local area is less likely to be an obstacle. In this case, the local area interpolation voting unit 550 does not use the approximated plane for voting. In addition, when the object is mixed with the background and the depth is dispersed, although it is a single object, it becomes an object that is significantly long in the depth direction from a short distance to a long distance. In this case, the local area interpolation voting unit 550 infers that the object is not correctly segmented by the local similar texture area segmentation unit 410. Therefore, the local similar texture area segmentation unit 410 re-segments the original image. Then, the local area interpolation voting unit 550 votes for the parallax of the re-segmented local area. However, the local area interpolation voting unit 550 sometimes does not vote for the parallax of the re-segmented local area. In this case, the object detection unit 600 detects the object only based on the number of votes for the parallax of the re-segmented local area voted by the parallax voting unit 300.

[0107] <Example 1 of Processing Using Three-Dimensional Point Group> Here, another example using a three-dimensional point group will be described. Fig.15 is a diagram for explaining processing of multiple texture regions included in the same object. Fig.15 , an example of the back side of the front vehicle 18 is shown.

[0108] The local similar texture region segmentation unit 410 (see Figure 7 ) Texture regions segmented from the original image may be segmented at different locations even if they are the same object. Similarly, the individual object regions segmented from the original image by the individual object region segmentation unit 430 may be included in the same object even if they are at different locations. For example, inside the front vehicle 18, the same texture is segmented into segmented regions 120 and 130 as texture regions or individual object regions. Therefore, there is a possibility that the segmented segmented regions 120 and 130 are segmented as different objects and vote for erroneous parallax.

[0109] Therefore, when the additional disparity voting unit (additional disparity voting unit 500) can confirm that multiple texture areas or multiple independent object areas are included in the same object (the preceding vehicle 18) through the distribution of the three-dimensional point group, it votes the effective disparity of the same object (the preceding vehicle 18) in the UD voting space. For example, the local area interpolation voting unit 550 or the independent obstacle interpolation voting unit 540 confirms the distribution of the three-dimensional point group relative to the segmented areas 120 and 130, and confirms whether the segmented areas 120 and 130 are included in the same preceding vehicle 18. When the local area interpolation voting unit 550 or the independent obstacle interpolation voting unit 540 can confirm that the segmented areas 120 and 130 are included in the same preceding vehicle 18 based on the distribution of the three-dimensional point group, the segmented areas 120 and 130 are voted as valid disparities of one object in the UD voting space. Through this processing, even if there are separate independent object areas inside an object, they are voted as the disparity of one object in the UD voting space.

[0110] On the other hand, when the local area interpolation voting unit 550 or the individual obstacle interpolation voting unit 540 can confirm that the segmented areas 120 and 130 are not included in the same preceding vehicle 18 , the segmented areas 120 and 130 are respectively treated as different objects and the parallax is voted in the UD voting space.

[0111] <Example 2 of Processing Using Three-Dimensional Point Group> In Example 1 of the processing using the above-mentioned three-dimensional point group, the local area interpolation voting unit 550 can also estimate the plane or curved surface of the segmented areas 120 and 130 when the segmented areas 120 and 130 are texture areas of the same texture contained in the same object. In this case, the additional parallax voting unit (additional parallax voting unit 500) estimates the plane or curved surface of the texture area, and when it can be confirmed that multiple texture areas are contained in the same object (front vehicle 18) based on the number and distance of the three-dimensional point groups existing at positions far away from the estimated plane or curved surface, the parallax that is valid for the same object (front vehicle 18) is voted in the UD voting space. The following processing is the same as Example 1 of the processing using the three-dimensional point group, so the description is omitted. By utilizing Example 2 of the processing using the three-dimensional point group, even if the texture areas are divided in separate places, if it can be confirmed that they are the same object, the parallax is voted between the texture areas located in separate places as the same object. Therefore, the object detection unit 600 can correctly detect the object.

[0112] <Example 3 of Processing Using Three-Dimensional Point Group> Fig.16 This is a diagram for explaining a process of confirming that the size of an individual object region is within a predetermined range. Fig.16 An example of the front vehicle 18 traveling at an angle is shown.

[0113] The individual object region segmentation unit 430 (see Figure 7 ) The entire front car 18 is segmented from the original image as an object into separate object regions, and a label is attached as "car1". However, if the segmented separate object region includes an object with an incorrect size as a separate object, the disparity of the separate object region is not suitable for voting.

[0114] Therefore, the additional disparity voting unit (additional disparity voting unit 500) votes the disparity that is valid as a separate object (front vehicle 18) in the UD voting space when it can be confirmed that the size of the separate object area exists as a separate object (vehicle) through the distribution of the three-dimensional point group. For example, if the separate object is a vehicle, it is confirmed whether the size of the vehicle is within a range of 3m in width and 10m in depth. The separate obstacle interpolation voting unit 540 votes the disparity of the separate object area as an object in the UD voting space when it can be confirmed that the size of the vehicle is within a specified range of 3m in width and 10m in depth. Through this process, the disparity of the separate object area of ​​appropriate size is voted in the UD voting space.

[0115] On the other hand, if the individual object is not within the specified range, the individual obstacle interpolation voting unit 540 can confirm that the background in addition to the object is also acquired into the individual object region. In this case, the individual obstacle interpolation voting unit 540 confirms the result and instructs the individual object region segmentation unit 430 to re-segment, and obtains the re-segmented individual object region. Thereafter, the individual obstacle interpolation voting unit 540 interpolates the disparity of the re-segmented individual object region.

[0116] The size range of the independent object region confirmed by the independent obstacle interpolation voting unit 540 varies depending on the type of object. For example, if it is an independent object region with a pedestrian label, it is sufficient to confirm whether the independent object region is within the range of 1m in width and 2m in height.

[0117] <Example of Processing of External Recognition Device> Next, an example of processing by the external recognition device 1 will be described. Fig.17 is a flowchart showing an example of the processing of the external recognition device 1. Here, Figure 1 The processing of each unit is explained using the functional blocks shown.

[0118] First, the external information acquisition unit 100 acquires external information from a stereo camera or the like (S1). Next, the disparity generation unit 200 generates disparity based on the external information (S2). Next, the disparity voting unit 300 votes for disparity in the UD voting space (S3).

[0119] Next, the texture region segmentation unit 400 segments the original image into texture regions based on the external information (S4). In addition, the processing of the texture region segmentation unit 400 may be performed in parallel with the processing of steps S2 and S3 after acquiring the external information in step S1.

[0120] Next, the additional disparity voting unit 500 votes for additional disparity based on the texture area in the UD voting space where the disparity was voted by the disparity voting unit 300 (S5). Next, the object detection unit 600 detects an object based on the disparity voted in the UD voting space (S6).

[0121] Then, the alarm control unit 700 controls the alarm when the object detected by the object detection unit 600 is an obstacle (S7). Through this alarm control, the vehicle can avoid the obstacle or stop in front of the obstacle. In addition, if the object detected by the object detection unit 600 is not an obstacle, the alarm control is not performed. In addition, the processing of the external recognition device 1 is repeated from step S1 at a predetermined time interval.

[0122] In the external recognition device 1 of one embodiment described above, when generating the UD voting space from the disparity image, the disparity is voted based on the texture area divided for each object, thereby making the recognition of each object in the UD voting space clear. Therefore, it is possible to suppress the non-detection or false detection of objects that were previously difficult to identify, such as roadside or black vehicles, and low-height dropped objects, and to achieve stable detection of objects, that is, robust detection.

[0123] The previous method is to add a label of the object category to the result of region segmentation. Therefore, when the object segmented by the region is larger than the original object, or the segmentation is performed because a part of the original object is unclear, thereby obtaining an erroneous result, it is possible that the vehicle is stopped urgently, or the vehicle continues to travel as an obstacle that does not hinder travel. On the other hand, in the method of this embodiment, before the object detection unit 600 detects the object, the additional parallax processing of the additional parallax voting unit 500 is added. Therefore, after voting the parallax according to the distance to the object, the object detection unit 600 accurately detects and analyzes the shape of the object. Then, based on the result of the object detection, the automatic driving control of the vehicle is performed. Therefore, according to the method of this embodiment, the vehicle will no longer be stopped urgently in front of an obstacle. In addition, according to the method of this embodiment, there is no possibility that the vehicle will continue to travel in a state where it is mistakenly identified as an object that does not hinder travel.

[0124] In addition, in the previous method, there are sometimes objects that have not been detected. On the other hand, in the method of this embodiment, even if an object has not been detected in the previous method, the object detection unit 600 can correctly detect the object by interpolating the parallax by the additional parallax voting unit 500 and performing additional voting on the object.

[0125] The present invention is not limited to the above-described embodiment, and various other application examples and modified examples can of course be adopted without departing from the gist of the present invention described in the claims. For example, the above-mentioned embodiment is to explain the present invention in detail and specifically in order to easily understand the embodiment of the device, and is not necessarily limited to the embodiment having all the described structures. In addition, for a part of the structure of this embodiment, other structures may be added, deleted, or replaced. In addition, the control lines and information lines are those that are considered necessary for the purpose of explanation, and not all control lines and information lines are necessarily shown in the product. In reality, it can be considered that almost all components are connected to each other. Explanation of symbols

[0126] 1…external recognition device, 100…external information acquisition unit, 110…camera, 200…disparity generation unit, 300…disparity voting unit, 400…texture region segmentation unit, 410…local similar texture region segmentation unit, 420…semantic region segmentation unit, 430…single object region segmentation unit, 500…additional disparity voting unit, 510…roadside additional voting unit, 520…road surface paint subtraction unit, 530…category obstacle interpolation voting unit, 540…single obstacle interpolation voting unit, 550…local region interpolation voting unit, 600…object detection unit, 700…alarm control unit.

Claims

1. An external identification device, characterized in that: have: a texture region segmentation unit that segments the external information into a plurality of texture regions according to a texture determined based on the external information acquired from the external information acquisition unit; a parallax generating unit configured to generate a parallax corresponding to an external depth distance based on the external information; a disparity voting unit that votes for a disparity corresponding to the depth distance of the object determined based on the external information in a voting space having the external depth distance on one axis; an additional parallax voting unit configured to vote for an additional parallax based on the texture area in the voting space that voted for the parallax; as well as An object detection unit detects the object based on the number of votes of the parallax voted in the voting space, thereby recognizing the exterior.

2. The external identification device according to claim 1, characterized in that: The additional disparity voting unit adds weighted disparity based on the texture area.

3. The external identification device according to claim 2, characterized in that: The additional parallax voting unit interpolates the parallax missing portion existing in the part corresponding to the texture area of ​​the parallax image having the depth distance on one axis using the parallax existing on the axial direction of the depth distance of the parallax image, and votes for the additional parallax according to the texture area after the parallax missing portion is interpolated.

4. The external identification device according to claim 2, characterized in that: The additional parallax voting unit interpolates the parallax missing part existing in the portion corresponding to the texture area of ​​the parallax image having the depth distance on one axis with the parallax around the parallax missing part existing in the same object as the object in which the parallax missing part exists, and votes for the additional parallax according to the texture area after the parallax missing part is interpolated.

5. The external identification device according to claim 3, characterized in that: The texture region segmentation unit comprises: a local similar texture region segmentation unit that segments the external information into the texture regions for each local similar texture; a semantic region segmentation unit that segments the external information into semantic regions according to the category of each of the objects; The individual object region dividing unit divides the external information into individual object regions for each of the objects.

6. The external identification device according to claim 5, characterized in that: The additional parallax voting unit includes a subtraction unit that reduces the number of votes for the parallax that votes for the road surface marking.

7. The external identification device according to claim 5, characterized in that: The additional parallax voting unit votes for parallaxes that are effective for the same object in the voting space when it can be confirmed from the distribution of the three-dimensional point group that a plurality of the texture regions or a plurality of the independent object regions are included in the same object.

8. The external identification device according to claim 5, characterized in that: The additional parallax voting unit estimates the plane or surface of the texture area, and when it can be confirmed that multiple texture areas are included in the same object based on the number and distance of three-dimensional point groups existing at positions far away from the estimated plane or surface, the additional parallax voting unit votes as valid parallax for the same object into the voting space.

9. The external identification device according to claim 5, characterized in that: The additional parallax voting unit votes the parallax effective as the independent object in the voting space when it can be confirmed from the distribution of the three-dimensional point group that the independent object region exists as the independent object.

10. The external identification device according to claim 5, characterized in that: The voting space has, on the other axis, an angle at which the external information acquisition unit can acquire the external information, or a lateral position of an image when the external information is an image.

11. The external identification device according to claim 5, characterized in that: The external information acquisition unit acquires information detected by the sensing device and information detected by other sensing devices as external information.

12. The external identification device according to claim 11, characterized in that: The sensing device is a first camera, and the other sensing device is a second camera arranged in a horizontal direction of the first camera. The external information acquisition unit acquires external images captured by the first camera and the second camera as the external information.

13. The external identification device according to claim 11, characterized in that: The sensing device is a monocular camera, and the other sensing device is a measuring device that measures the distance to the object by irradiating electromagnetic waves to the object to obtain the distribution of a three-dimensional point group. The external information acquisition unit acquires, as the external information, an external image captured by the monocular camera and a distribution of the three-dimensional point group output from the measurement device.

14. An external identification method, which is performed by an external identification device, characterized in that: include: The external information acquisition department acquires external information; According to the texture of each object represented by the external information, the external information is divided into a plurality of texture regions; A process of generating a parallax corresponding to an external depth distance based on the external information; In a voting space having the external depth distance on one axis, voting is performed on the parallax corresponding to the depth distance of the object determined according to the external information; an additional disparity voting unit configured to vote for an additional disparity based on the texture area in the voting space that voted for the disparity; as well as A processing unit detects the object based on the number of votes of the parallax voted in the voting space, thereby recognizing the exterior.

Images