Image processing device and image processing method

Abstract

To provide an image processing apparatus configured to improve the accuracy of determining whether a three-dimensional object is a vehicle or not.SOLUTION: An image processing apparatus according to one embodiment of the present disclosure includes: a first area setting unit which sets a first image area by detecting a three-dimensional object, on the basis of a distance value included in a distance image generated based on a stereo image; a second area setting unit which sets a second image area by estimating a vehicle degree which indicates a degree that the three-dimensional object is a vehicle, on the basis of one of a left image and a right image included in the stereo image; a comparison unit which compares the first image area with the second image area; and a determination unit which determines whether the three-dimensional object is a vehicle or not on the basis of the vehicle degree and comparison results of the comparison unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image processing device that analyzes an imaged three-dimensional object, and an image processing method used in such an image processing device. [Background technology]

[0002] Some image processing devices mounted on vehicles such as automobiles analyze captured images obtained by an image sensor. For example, Patent Document 1 discloses a vehicle detection method that includes a search radar and an image sensor, and predicts a rectangular area surrounding an image of a vehicle included in a captured image based on the detection result of the search radar. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2005-141517 A Summary of the Invention [Problem to be solved by the invention]

[0004] It is desirable for an image processing device mounted on a vehicle to be able to accurately determine whether a detected three-dimensional object is a vehicle, and further improvements in the determination accuracy are anticipated.

[0005] It is desirable to provide an image processing device and an image processing method that can improve the accuracy of determining whether a three-dimensional object is a vehicle. [Means for solving the problem]

[0006] An image processing device according to an embodiment of the present disclosure includes a first region setting unit, a second region setting unit, a comparison unit, and a determination unit. The first region setting unit is configured to set a first image region by detecting a three-dimensional object based on a distance image generated based on a stereo image. The second region setting unit is configured to set a second image region by estimating a vehicle degree indicating the degree to which the three-dimensional object is a vehicle based on one of a left image and a right image included in the stereo image. The comparison unit is Based on the size of the first image area, the size of the second image area, and the relative positional relationship between the first image area and the second image area, The first image region and the second image region are compared. The determination unit is configured to determine whether the three-dimensional object is a vehicle based on the vehicle degree and a processing result of the comparison unit.

[0007] An image processing method according to an embodiment of the present disclosure includes: An image processing method executed in an image processing device, comprising: setting a first image area by detecting a three-dimensional object based on a distance image generated based on a stereo image, and setting a second image area by estimating a vehicle degree indicating a degree to which the three-dimensional object is a vehicle based on one of a left image and a right image included in the stereo image; Based on the size of the first image area, the size of the second image area, and the relative positional relationship between the first image area and the second image area, The method includes performing a comparison process of comparing the first image area with the second image area, and determining whether the three-dimensional object is a vehicle based on the vehicle degree and a processing result of the comparison process. Effect of the Invention

[0008] According to an image processing device and an image processing method according to an embodiment of the present disclosure, it is possible to improve the accuracy of determining whether a three-dimensional object is a vehicle. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating an example of a configuration of an image processing device according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is an explanatory diagram illustrating an example of a stereo image shown in FIG. [Diagram 3] 2 is an explanatory diagram illustrating an example of an operation of the three-dimensional object detection unit illustrated in FIG. 1. [Figure 4] 2 is an explanatory diagram illustrating an operation example of a vehicle search unit illustrated in FIG. 1. [Diagram 5] FIG. 2 is an explanatory diagram illustrating an example of a solid object region and a vehicle region. [Figure 6] FIG. 11 is an explanatory diagram illustrating another example of a solid object region and a vehicle region. [Figure 7] FIG. 2 is an explanatory diagram illustrating an example of the positional relationship between a solid object region and a vehicle region. [Figure 8] FIG. 11 is an explanatory diagram illustrating another example of the positional relationship between the three-dimensional object region and the vehicle region. [Figure 9] FIG. 11 is an explanatory diagram illustrating another example of the positional relationship between the three-dimensional object region and the vehicle region. [Figure 10] 4 is a flowchart illustrating an example of an operation of a vehicle determination unit illustrated in FIG. 1. [Figure 11] FIG. 13 is a block diagram illustrating an example of the configuration of an image processing device according to a modified example. [Figure 12] 12 is a flowchart illustrating an example of an operation of a vehicle determination unit illustrated in FIG. 11. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0011] <Embodiment Mode> [Configuration example] 1 shows an example of the configuration of an image processing device (image processing device 1) according to an embodiment. Note that an image processing method according to the embodiment of the present disclosure is embodied in this embodiment, and will be described together. The image processing device 1 includes a stereo camera 11 and a processing unit 20. The image processing device 1 is mounted on a vehicle 10 such as an automobile.

[0012] The stereo camera 11 is configured to capture an image of the front of the vehicle 10, thereby generating a pair of images (a left image PL and a right image PR) having a parallax between them. The stereo camera 11 has a left camera 11L and a right camera 11R. Each of the left camera 11L and the right camera 11R includes a lens and an image sensor. In this example, the left camera 11L and the right camera 11R are disposed in the vehicle 10, near the upper part of the windshield of the vehicle 10, and spaced apart by a predetermined distance in the width direction of the vehicle 10. The left camera 11L and the right camera 11R perform imaging operations in synchronization with each other. The left camera 11L generates a left image PL, and the right camera 11R generates a right image PR. The left image PL and the right image PR constitute a stereo image PIC. The stereo camera 11 performs imaging operations at a predetermined frame rate (for example, 60 [fps]) to generate a series of stereo images PIC.

[0013] 2 shows an example of a stereo image PIC, where (A) shows an example of a left image PL and (B) shows an example of a right image PR. In this example, a leading vehicle 90 is traveling ahead of the vehicle 10 on a road on which the vehicle 10 is traveling. The left camera 11L captures an image of the leading vehicle 90 to generate a left image PL, and the right camera 11R captures an image of the leading vehicle 90 to generate a right image PR. The stereo camera 11 is adapted to generate a stereo image PIC including such a left image PL and a right image PR.

[0014] The processing unit 20 (FIG. 1) is configured to recognize a vehicle ahead of the vehicle 10 based on the stereo image PIC supplied from the stereo camera 11. In the vehicle 10, for example, based on the information about the three-dimensional object recognized by the processing unit 20, the vehicle 10 can perform driving control of the vehicle 10, such as AEB (Automatic Emergency Braking) or ACC (Adaptive Cruise Control), or display the information about the recognized vehicle on a console monitor. The processing unit 20 is configured, for example, by a CPU (Central Processing Unit) that executes a program, a RAM (Random Access Memory) that temporarily stores processing data, and a ROM (Read Only Memory) that stores the program. The processing unit 20 has a distance image generating unit 21, a three-dimensional object detecting unit 22, a continuous structure estimating unit 23, a vehicle searching unit 24, an image area comparing unit 25, and a vehicle determining unit 26.

[0015] The distance image generating unit 21 is configured to generate a distance image PZ by performing a predetermined image processing including a stereo matching process and a filtering process based on the left image PL and the right image PR included in the stereo image PIC. The distance image generating unit 21 performs a stereo matching process by identifying corresponding points including two image points that correspond to each other based on the left image PL and the right image PR. The pixel value of each pixel of the distance image PZ is a parallax value, and is, for example, the difference between the coordinate value of the horizontal coordinate of the image point in the left image PL and the coordinate value of the horizontal coordinate of the image point in the right image PR. This parallax value corresponds to a distance value to a point corresponding to each pixel in a three-dimensional real space. Note that this is not limited to this, and for example, each of the multiple pixel values ​​may be a distance value indicating the distance to a point corresponding to each pixel in a three-dimensional real space. The distance image generating unit 21 is configured to supply the generated distance image PZ to the three-dimensional object detection unit 22 and the continuous structure estimation unit 23.

[0016] The three-dimensional object detection unit 22 is configured to detect a three-dimensional object based on the distance image PZ. The three-dimensional object detection unit 22 is configured to set a three-dimensional object region Robj in an image region in the distance image PZ that corresponds to the detected three-dimensional object.

[0017] FIG. 3 shows an example of a three-dimensional object region Robj in the distance image PZ detected by the three-dimensional object detection unit 22. Although the distance image PZ is an image of parallax values, in FIG. 3, the three-dimensional object itself is depicted for convenience of explanation. In the distance image PZ, there are continuous parallax values ​​in an image region corresponding to a three-dimensional object such as a preceding vehicle 90, a guardrail, a wall, a building, or a person. The three-dimensional object detection unit 22 detects a three-dimensional object by using such continuous parallax values ​​included in the distance image PZ. Then, the three-dimensional object detection unit 22 sets a three-dimensional object region Robj in an image region corresponding to the detected three-dimensional object in the distance image PZ. When an image region with continuous parallax values ​​is wide, such as a guardrail or a wall, the three-dimensional object detection unit 22 sets the three-dimensional object region Robj within a range not exceeding a predetermined size according to the size of the vehicle assumed when the vehicle is present at a distance corresponding to the parallax value. The three-dimensional object detection unit 22 then supplies information about the set three-dimensional object region Robj and information about the disparity value in that three-dimensional object region Robj to the continuous structure estimation unit 23, the image area comparison unit 25, and the vehicle determination unit 26.

[0018] The continuous structure estimation unit 23 is configured to estimate the degree to which the three-dimensional object detected by the three-dimensional object detection unit 22 is a continuous structure based on the distance image PZ. A continuous structure is, for example, a structure such as a guardrail or a wall that is connected over a relatively long distance along the road. The continuous structure estimation unit 23 estimates the degree to which the three-dimensional object is such a continuous structure (continuous structure degree D) and supplies information on the continuous structure degree D to the vehicle determination unit 26.

[0019] The continuous structure estimation unit 23 estimates the continuity structure degree D by, for example, checking whether the disparity values ​​are continuous in the three-dimensional object region Robj set by the three-dimensional object detection unit 22 and the image regions to the left and right of the three-dimensional object region Robj. The continuity structure degree D indicates a higher value as the possibility that the three-dimensional object is a continuous structure increases.

[0020] Specifically, for example, as shown in Fig. 3, when the three-dimensional object is a leading vehicle 90, the disparity values ​​are not continuous in the three-dimensional object area Robj and the image areas on the left and right of the three-dimensional object area Robj. That is, the disparity value in the three-dimensional object area Robj corresponds to the distance to the leading vehicle 90, and the disparity values ​​in the image areas on the left and right of the three-dimensional object area Robj correspond to, for example, the distance to the road on which the vehicle 10 is traveling, and are therefore smaller than the disparity value in the three-dimensional object area Robj. In this case, the continuous structure estimation unit 23 estimates that the degree to which the three-dimensional object is a continuous structure (continuous structure degree D) is low.

[0021] Also, for example, when the solid object is a guardrail or a wall, the disparity values ​​are continuous in the solid object area Robj and the image areas on the left and right of the solid object area Robj. That is, as described above, the solid object area Robj is set within a range not exceeding a predetermined size according to the size of the vehicle assumed when the vehicle is at the position corresponding to the disparity value, so the solid object area Robj is set, for example, in a part of the area corresponding to the guardrail or the wall. Therefore, in this case, the disparity values ​​are continuous in the solid object area Robj and the image areas on the left and right of the solid object area Robj, so the continuous structure estimation unit 23 estimates that the degree to which the solid object is a continuous structure (continuous structure degree D) is high.

[0022] The vehicle search unit 24 is configured to search for a vehicle based on the image P, which is one of the left image PL and the right image PR, by using a machine learning technique.

[0023] FIG. 4 is a schematic diagram showing an example of an operation of the vehicle search unit 24. The image P includes an image of a preceding vehicle 90 traveling ahead of the vehicle 10. The vehicle search unit 24 sequentially sets a plurality of rectangular processing target regions Rp in the image P, for example, by gradually changing the position and size. Then, the vehicle search unit 24 calculates a vehicle score SC in each processing target region Rp using a machine learning technique. The vehicle score SC indicates a higher value as the image of the processing target region Rp includes vehicle features. For example, when the vehicle score SC of the processing target region Rp having the highest vehicle score SC among the plurality of processing target regions Rp is equal to or higher than a predetermined score, the vehicle search unit 24 sets the processing target region Rp as the vehicle region Rv. Then, the vehicle search unit 24 supplies information about the vehicle region Rv and the vehicle score SC to the image region comparison unit 25 and the vehicle determination unit 26.

[0024] The image area comparison unit 25 is configured to compare the three-dimensional object area Robj set by the three-dimensional object detection unit 22 with the vehicle area Rv set by the vehicle search unit 24.

[0025] 5 and 6 show an example of the solid object region Robj and the vehicle region Rv. FIG. 5 shows a case where the solid object is a leading vehicle 90, and FIG. 6 shows a case where the solid object is a guard rail.

[0026] 5, the three-dimensional object detection unit 22 sets a three-dimensional object area Robj in an area corresponding to the preceding vehicle 90, and the vehicle search unit 24 sets a vehicle area Rv in an area corresponding to the preceding vehicle 90. When the three-dimensional object is a vehicle, for example, as shown in FIG. 5, the position and size of the three-dimensional object area Robj are substantially the same as the position and size of the vehicle area Rv.

[0027] In the example of FIG. 6, the three-dimensional object detection unit 22 sets the three-dimensional object area Robj in an area corresponding to the guardrail 91, the pole 92, and the sign 93, and the vehicle search unit 24 sets the vehicle area Rv in an area corresponding to a part of the guardrail 91 and a part of the pole 92. That is, in this example, the three-dimensional object is far away, and the image is rough, so the vehicle search unit 24 determines that the image pattern shown by the part of the guardrail 91 and the part of the pole 92 has a vehicle-like characteristic, increases the vehicle score SC, and sets the vehicle area Rv to the part of the guardrail 91 and the part of the pole 92. When the three-dimensional object includes a continuous structure, the three-dimensional object area Robj may be large as shown in FIG. 6. In particular, in this example, the three-dimensional object is far away, and the accuracy of the parallax value is low. In addition, the pole 92 and the sign 93 are near the guardrail 91, which is a continuous structure, so the image area having the same parallax value is wide. As a result, the three-dimensional object detection unit 22 sets the three-dimensional object area Robj larger than the vehicle area Rv.

[0028] 7 to 9 show an example of the relationship between the three-dimensional object region Robj and the vehicle region Rv in the case where the three-dimensional object includes a continuous structure.

[0029] In the example of Fig. 7, the vehicle region Rv is set so as to straddle the lower side of the solid object region Robj, similarly to the example of Fig. 6. In other words, in this example, the vehicle region Rv is included in the solid object region Robj in the horizontal direction (x direction). Note that this is not limited thereto, and for example, the vehicle region Rv may be set so as to straddle the upper side of the solid object region Robj.

[0030] In the example of Fig. 8, the vehicle region Rv is set so as to straddle the right side of the solid object region Robj. In other words, in this example, the vehicle region Rv is included in the solid object region Robj in the vertical direction (y direction). Note that this is not limited to this, and for example, the vehicle region Rv may be set so as to straddle the left side of the solid object region Robj.

[0031] In the example of FIG. 9, the vehicle region Rv is enclosed within the solid object region Robj. That is, in this example, in the horizontal direction (x-direction), the vehicle region Rv is included in the solid object region Robj, and similarly, in the vertical direction (y-direction), the vehicle region Rv is included in the solid object region Robj.

[0032] The image region comparison unit 25 compares the solid object region Robj and the vehicle region Rv by checking whether the solid object region Robj and the vehicle region Rv satisfy a predetermined determination condition C. This determination condition C includes, for example, that in at least one of the horizontal direction (x-direction) and the vertical direction (y-direction), the vehicle region Rv is included in the solid object region Robj, and the overlap ratio A obtained by dividing the width Wv of the vehicle region Rv by the width Wobj of the solid object region Robj in that direction is smaller than a predetermined threshold TH (A = Wv / Wobj < TH). Satisfying this determination condition C indicates that the solid object is likely to include a continuous structure.

[0033] For example, as shown in FIGS. 7 and 9, in the horizontal direction (x-direction), the vehicle region Rv may be included in the solid object region Robj and the overlap ratio A (overlap ratio Ax) may be smaller than a predetermined threshold THx (Ax = Wvx / Wobjx < THx). Also, for example, as shown in FIGS. 8 and 9, in the vertical direction (y-direction), the vehicle region Rv may be included in the solid object region Robj and the overlap ratio A (overlap ratio Ay) may be smaller than a predetermined threshold Thy (Ay = Wvy / Wobjy < Thy).

[0034] The image region comparison unit 25 compares the solid object region Robj and the vehicle region Rv by checking whether the solid object region Robj and the vehicle region Rv satisfy such a determination condition C. Then, the image region comparison unit 25 supplies this comparison result to the vehicle determination unit 26.

[0035] The vehicle determination unit 26 is configured to determine whether or not a three-dimensional object is a vehicle based on the detection results of the three-dimensional object detection unit 22, the continuous structure degree D obtained by the continuous structure estimation unit 23, the search results of the vehicle search unit 24, the comparison results of the image area comparison unit 25, and driving information of the vehicle 10 obtained from a vehicle control device (not shown) of the vehicle 10.

[0036] Here, the three-dimensional object detection unit 22 corresponds to a specific example of the "first region setting unit" in the present disclosure. The three-dimensional object region Robj corresponds to a specific example of the "first image region" in the present disclosure. The vehicle search unit 24 corresponds to a specific example of the "second region setting unit" in the present disclosure. The vehicle region Rv corresponds to a specific example of the "second image region" in the present disclosure. The vehicle score SC corresponds to a specific example of the "vehicle degree" in the present disclosure. The image region comparison unit 25 corresponds to a specific example of the "comparison unit" in the present disclosure. The vehicle determination unit 26 corresponds to a specific example of the "determination unit" in the present disclosure. The determination condition C corresponds to a specific example of the "determination condition" in the present disclosure. The threshold value TH corresponds to a specific example of the "threshold value" in the present disclosure. The continuous structure estimation unit 23 corresponds to a specific example of the "estimation unit" in the present disclosure. The continuous structure degree D corresponds to a specific example of the "continuous structure degree" in the present disclosure.

[0037] [Actions and Actions] Next, the operation and function of the image processing device 1 of this embodiment will be described.

[0038] (Overview of overall operation) First, an overview of the overall operation of the image processing device 1 will be described with reference to FIG. 1. The stereo camera 11 captures an image in front of the vehicle 10 to generate a stereo image PIC including a left image PL and a right image PR having a parallax from each other. The distance image generating unit 21 generates a distance image PZ based on the left image PL and the right image PR included in the stereo image PIC. The three-dimensional object detecting unit 22 detects a three-dimensional object based on the distance image PZ to set a three-dimensional object area Robj. The continuous structure estimating unit 23 estimates the degree to which the three-dimensional object detected by the three-dimensional object detecting unit 22 is a continuous structure (continuous structure degree D) based on the distance image PZ. The vehicle searching unit 24 searches for a vehicle using a machine learning technique based on the image P, which is one of the left image PL and the right image PR, to set a vehicle area Rv. The image area comparing unit 25 compares the three-dimensional object area Robj with the vehicle area Rv. The vehicle determination unit 26 determines whether the three-dimensional object is a vehicle based on the detection results of the three-dimensional object detection unit 22, the continuous structure degree D obtained by the continuous structure estimation unit 23, the search results of the vehicle search unit 24, the comparison results of the image area comparison unit 25, and the driving information of the vehicle 10 obtained from a vehicle control device (not shown) of the vehicle 10.

[0039] (Details of operation) When searching for a vehicle, the vehicle search unit 24 sequentially sets a plurality of processing target areas Rp in the image P, and calculates the vehicle score SC by using machine learning technology to check whether the image in each processing target area Rp includes vehicle characteristics. In this case, if the image in the processing target area Rp is not an image of a vehicle, but the image pattern has vehicle-like characteristics, the vehicle score SC will be high. For example, if the image in the processing target area Rp is an image of a continuous structure such as a guardrail or a wall, the image pattern may have vehicle-like characteristics, and in such a case, the vehicle score SC will be high.

[0040] In this way, if the continuous structure is erroneously determined to be a vehicle, a malfunction occurs in the driving control of the vehicle 10, such as AEB or ACC. Therefore, in the image processing device 1, the image area comparison unit 25 compares the three-dimensional object area Robj with the vehicle area Rv by checking whether the three-dimensional object area Robj and the vehicle area Rv satisfy the determination condition C. Then, the vehicle determination unit 26 determines whether the three-dimensional object is a vehicle based on the detection result of the three-dimensional object detection unit 22, the continuous structure degree D obtained by the continuous structure estimation unit 23, the search result of the vehicle search unit 24, and the comparison result of the image area comparison unit 25. In particular, when the three-dimensional object area Robj and the vehicle area Rv satisfy the determination condition C, the vehicle determination unit 26 determines that the three-dimensional object is highly likely to include a continuous structure, and can perform a vehicle determination suppression process B that makes it difficult to determine that the three-dimensional object is a vehicle. In the vehicle determination suppression process B, the vehicle determination unit 26 can make it difficult for the three-dimensional object to be determined to be a vehicle, for example, by correcting the vehicle score SC obtained by the vehicle search unit 24 to a lower score or by changing the threshold value for performing vehicle determination based on the vehicle score SC to a higher value. In this way, the image processing device 1 can increase the accuracy of determining whether a three-dimensional object is a vehicle.

[0041] FIG. 10 shows an example of a process in the vehicle determination section 26 for determining whether or not to perform the vehicle determination suppression process B.

[0042] First, the vehicle determination unit 26 checks whether or not a three-dimensional object in the three-dimensional object area Robj set by the three-dimensional object detection unit 22 is a distant three-dimensional object (step S101). Specifically, the vehicle determination unit 26 checks whether or not the distance to the three-dimensional object indicated by the disparity value in the three-dimensional object area Robj satisfies the following condition, thereby checking whether or not the three-dimensional object is a distant three-dimensional object. Threshold THmin < Distance < Threshold THmax If the three-dimensional object is a distant three-dimensional object, the vehicle search unit 24 is likely to erroneously determine that the continuous structure is a vehicle. The threshold values ​​THmin and THmax are the upper and lower limits of the distance at which the vehicle search unit 24 is likely to erroneously determine. The vehicle determination unit 26 checks whether this condition is met to determine whether the vehicle search unit 24 is likely to erroneously determine. If the object is not a distant three-dimensional object ("N" in step S101), this flow ends. That is, in this case, since the vehicle search unit 24 is unlikely to erroneously determine, the vehicle determination unit 26 ends this process without performing the vehicle determination suppression process B.

[0043] In step S101, if the object is a distant three-dimensional object ("Y" in step S101), the vehicle determination unit 26 checks whether the comparison result of the image area comparison unit 25 indicates that the vehicle area Rv is included in the three-dimensional object area Robj in the horizontal direction (x direction) and that the overlap rate Ax satisfies the determination condition C that is less than or equal to a predetermined threshold value THx (step S102).

[0044] In step S102, if the judgment condition C is not satisfied ("N" in step S102), the vehicle judgment unit 26 checks whether the comparison result of the image area comparison unit 25 indicates that the judgment condition C is satisfied, that is, the vehicle area Rv is included in the solid object area Robj in the vertical direction (y direction) and the overlap rate Ay is equal to or less than a predetermined threshold value THy (step S103). If the judgment condition C is not satisfied ("Y" in step S103), the process ends. That is, in this case, the judgment condition C in step S102 and the judgment condition C in step S103 are not satisfied, so that the three-dimensional object is unlikely to include a continuous structure, and the vehicle judgment unit 26 ends the process without performing the vehicle judgment suppression process B.

[0045] If the judgment condition C of step S102 is satisfied ("Y" in step S102), or if the judgment condition C of step S103 is satisfied ("Y" in step S103), the vehicle judgment unit 26 checks the contiguous structure degree D estimated by the contiguous structure estimation unit 23 (step S104). In this example, the contiguous structure estimation unit 23 estimates the contiguous structure degree D at three levels ("high", "medium", "low").

[0046] In step S104, if the continuity degree D is at a high level ("high" in step S104), the vehicle determination unit 26 checks whether the time (TTC: Time to Collision) until the vehicle 10 collides with the three-dimensional object is 5 seconds or more (step S105). Specifically, the vehicle determination unit 26 calculates the time until the vehicle 10 collides with the three-dimensional object based on, for example, the distance to the three-dimensional object indicated by the disparity value in the three-dimensional object region Robj and information on the vehicle speed of the vehicle 10 acquired from a vehicle control device (not shown) of the vehicle 10, and checks whether this time is 5 seconds or more. If this time is not 5 seconds or more ("N" in step S105), this process ends. That is, in this case, since the time until the collision is short, it does not matter much whether the three-dimensional object is a vehicle. In addition, even if the three-dimensional object is a vehicle, the continuity degree D may be erroneously determined to be at a high level by the continuity estimation unit 23. Therefore, the vehicle determination unit 26 does not perform the vehicle determination suppression process B and ends this process.

[0047] In step S104, if the degree of continuous structure D is at a medium level ("medium" in step S104), the vehicle determination unit 26 checks whether the time (TTC) until the vehicle 10 collides with the three-dimensional object is 7 seconds or more (step S106). If this time is not 7 seconds or more ("N" in step S106), this process ends.

[0048] In step S105, if the time until vehicle 10 collides with the three-dimensional object is 5 seconds or more ("Y" in step S105), or if the time until vehicle 10 collides with the three-dimensional object is 7 seconds or more ("Y" in step S106), vehicle determination unit 26 performs vehicle determination suppression process B to make it less likely that the three-dimensional object is determined to be a vehicle (step S107).

[0049] This is the end of the flow. In this way, the vehicle determination unit 26 repeats this flow to monitor whether the three-dimensional object is likely to include a continuous structure, and when the three-dimensional object is likely to include a continuous structure, performs a vehicle determination suppression process B that makes it difficult to determine that the three-dimensional object is a vehicle. This allows the image processing device 1 to increase the accuracy of determining whether the three-dimensional object is a vehicle.

[0050] In this way, the image processing device 1 is provided with an image area comparison unit 25 that compares the three-dimensional object area Robj with the vehicle area Rv, and the vehicle determination unit 26 determines whether or not the three-dimensional object is a vehicle based on the vehicle score SC supplied from the vehicle search unit 24 and the comparison result in the image area comparison unit 25. This allows the vehicle determination unit 26 to grasp, for example, whether or not the three-dimensional object is likely to be a continuous structure such as a guardrail or a wall based on the comparison result in the image area comparison unit 25. As a result, the image processing device 1 can improve the accuracy of determining whether or not the three-dimensional object is a vehicle.

[0051] In addition, in the image processing device 1, the image area comparison unit 25 compares the solid object area Robj with the vehicle area Rv by checking whether the solid object area Robj and the vehicle area Rv satisfy the judgment condition C. The judgment condition C includes that the vehicle area Rv is included in the solid object area Robj in at least one of the horizontal direction (x direction) and vertical direction (y direction), and that the overlap rate A obtained by dividing the width of the vehicle area Rv in that direction by the width of the solid object area Robj in that direction is smaller than a predetermined threshold value TH. Thereby, for example, the vehicle judgment unit 26 can grasp whether the solid object is likely to be a continuous structure such as a guardrail or a wall based on the comparison result in the image area comparison unit 25. As a result, the image processing device 1 can improve the accuracy of judging whether the solid object is a vehicle.

[0052] Furthermore, in the image processing device 1, the vehicle determination unit 26 performs a vehicle determination suppression process that makes it difficult to determine that a three-dimensional object is a vehicle when the determination condition C is satisfied, so that the vehicle determination suppression process can be performed when the three-dimensional object is highly likely to be a continuous structure such as a guardrail or a wall. Therefore, in the image processing device 1, it is possible to increase the accuracy of determining whether a three-dimensional object is a vehicle.

[0053] Furthermore, the image processing device 1 is provided with a continuous structure estimation unit 23 that estimates a continuous structure degree D that indicates the degree to which the three-dimensional object is a continuous structure, and when the judgment condition C is satisfied, the vehicle judgment unit 26 judges whether the three-dimensional object is a vehicle or not based on the vehicle score SC and the continuous structure degree D. This allows the image processing device 1 to check the degree to which the three-dimensional object is a continuous structure in more detail, thereby improving the accuracy of judging whether the three-dimensional object is a vehicle or not.

[0054] In addition, in the image processing device 1, in steps S105 and S106 shown in FIG. 10, it is determined whether or not to perform the vehicle determination suppression process B depending on the time (TTC) until the vehicle 10 collides with the three-dimensional object. As a result, for example, when the time until the vehicle 10 collides with the three-dimensional object is short, it is possible to not perform the vehicle determination suppression process B regardless of whether the three-dimensional object is a vehicle. In addition, the reference time until the vehicle 10 collides with the three-dimensional object when the continuity degree D is at a high level ("5 seconds" in step S105 of FIG. 10) is set shorter than the reference time until the vehicle 10 collides with the three-dimensional object when the continuity degree D is at a medium level ("5 seconds" in step S105 of FIG. 10). As a result, in the image processing device 1, when the three-dimensional object is highly likely to be a continuous structure, it is possible to easily perform the vehicle determination suppression process B, thereby improving the accuracy of determining whether the three-dimensional object is a vehicle.

[0055] [effect] As described above, in this embodiment, an image area comparison unit is provided that compares the three-dimensional object area with the vehicle area, and the vehicle determination unit determines whether or not a three-dimensional object is a vehicle based on the vehicle score supplied from the vehicle search unit and the comparison result in the image area comparison unit, thereby improving the accuracy of determining whether or not a three-dimensional object is a vehicle.

[0056] In this embodiment, the image area comparison unit compares the three-dimensional object area with the vehicle area by checking whether the three-dimensional object area and the vehicle area satisfy a judgment condition. The judgment condition includes that the vehicle area is included in the three-dimensional object area in at least one of the horizontal direction (x direction) and vertical direction (y direction), and that the overlap rate obtained by dividing the width of the vehicle area in that direction by the width of the three-dimensional object area in that direction is smaller than a predetermined threshold value. This can improve the accuracy of judging whether the three-dimensional object is a vehicle.

[0057] In this embodiment, the vehicle determination unit performs vehicle determination suppression processing that makes it less likely that a three-dimensional object is determined to be a vehicle when the determination conditions are satisfied, thereby improving the accuracy of determining whether a three-dimensional object is a vehicle.

[0058] In this embodiment, a continuous structure estimation unit is provided that estimates a continuous structure degree D that indicates the degree to which a three-dimensional object is a continuous structure, and a vehicle determination unit is configured to determine whether or not a three-dimensional object is a vehicle based on the vehicle score and the continuous structure degree when a determination condition is satisfied, thereby improving the accuracy of determining whether or not a three-dimensional object is a vehicle.

[0059] [Variation 1] In the above embodiment, as shown in Fig. 10, when at least one of the judgment condition C of step S102 and the judgment condition C of step S103 is satisfied, the vehicle judgment unit 26 is allowed to perform the vehicle judgment suppression process B, but this is not limited to this. Instead of this, for example, when both the judgment condition C of step S102 and the judgment condition C of step S103 are satisfied, the vehicle judgment unit 26 may be allowed to perform the vehicle judgment suppression process B. Also, for example, step S103 may be omitted, and when the judgment condition C of step S102 is satisfied, the vehicle judgment unit 26 may be allowed to perform the vehicle judgment suppression process B. Also, for example, step S102 may be omitted, and when the judgment condition C of step S103 is satisfied, the vehicle judgment unit 26 may be allowed to perform the vehicle judgment suppression process B. Also, for example, step S102 may be omitted, and when the judgment condition C of step S103 is satisfied, the vehicle judgment unit 26 may be allowed to perform the vehicle judgment suppression process B.

[0060] [Variation 2] In the above embodiment, the image area comparison unit 25 checks whether the overlap rate Ax is smaller than a predetermined threshold value THx in the horizontal direction (x direction), but for example, this threshold value THx may be changeable. Specifically, for example, the threshold value THx may change according to the distance to the three-dimensional object or according to the vehicle score SC.

[0061] Similarly, the image area comparison unit 25 checks whether the overlap rate Ay in the vertical direction (y direction) is smaller than a predetermined threshold value THy, but for example, this threshold value THy may be changeable. Specifically, for example, the threshold value THy may change according to the distance to the three-dimensional object or according to the vehicle score SC.

[0062] [Variation 3] In the above embodiment, the continuous structure estimation unit 23 is provided, but the present invention is not limited to this, and instead, for example, the continuous structure estimation unit 23 may be omitted. An image processing device 1A according to this modification will be described in detail below.

[0063] 11 shows an example of the configuration of an image processing device 1A. The image processing device 1A includes a processing unit 20A. The processing unit 20A includes a distance image generating unit 21, a three-dimensional object detecting unit 22, a vehicle searching unit 24, an image area comparing unit 25, and a vehicle determining unit 26A. That is, the processing unit 20A according to this modification is obtained by omitting the continuous structure estimating unit 23 from the processing unit 20 (FIG. 1) according to the above embodiment, and by changing the vehicle determining unit 26 to a vehicle determining unit 26A.

[0064] The vehicle determination unit 26A is configured to determine whether a three-dimensional object is a vehicle based on the detection result of the three-dimensional object detection unit 22, the search result of the vehicle search unit 24, the comparison result of the image area comparison unit 25, and driving information of the vehicle 10 obtained from a vehicle control device (not shown) of the vehicle 10.

[0065] FIG. 12 shows an example of a process for determining whether or not to perform the vehicle determination suppression process B in the vehicle determination unit 26A.

[0066] First, the vehicle determination unit 26 checks whether the three-dimensional object in the three-dimensional object area Robj set by the three-dimensional object detection unit 22 is a distant three-dimensional object (step S101). If the three-dimensional object is not a distant three-dimensional object ("N" in step S101), this flow ends.

[0067] In step S101, if the object is a distant three-dimensional object ("Y" in step S101), the vehicle determination unit 26 checks whether the comparison result of the image area comparison unit 25 indicates that the vehicle area Rv is included in the three-dimensional object area Robj in the horizontal direction (x direction) and that the overlap rate Ax satisfies the determination condition C that is less than or equal to a predetermined threshold value THx (step S102).

[0068] In step S102, if the judgment condition C is not satisfied ("N" in step S102), the vehicle judgment unit 26 checks whether the comparison result of the image area comparison unit 25 indicates that the judgment condition C is satisfied, that is, the vehicle area Rv is included in the solid object area Robj in the vertical direction (y direction) and the overlap rate Ay is equal to or less than a predetermined threshold value THy (step S103). If the judgment condition C is not satisfied ("Y" in step S103), the process ends.

[0069] If the judgment condition C of step S102 is satisfied ("Y" in step S102), or if the judgment condition C of step S103 is satisfied ("Y" in step S103), vehicle judgment unit 26A performs vehicle judgment suppression process B that makes it difficult to judge that the three-dimensional object is a vehicle (step S107). In other words, satisfying at least one of judgment condition C of step S102 and judgment condition C of step S103 indicates that the three-dimensional object is highly likely to be a continuous structure, so vehicle judgment unit 26A performs vehicle judgment suppression process B that makes it difficult to judge that the three-dimensional object is a vehicle. This is the end of this flow.

[0070] [Other variations] Also, two or more of these variations may be combined.

[0071] Although the present technology has been described above by giving embodiments and modifications, the present technology is not limited to these embodiments and the like, and various modifications are possible.

[0072] For example, in the above embodiment, the stereo camera 11 is configured to capture an image in front of the vehicle 10, but this is not limited thereto. For example, the stereo camera 11 may capture an image of the side or rear of the vehicle 10.

[0073] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained. [Explanation of symbols]

[0074] 1,1A...image processing device, 11...stereo camera, 11L...left camera, 11R...right camera, 20,20A...processing unit, 21...distance image generation unit, 22...three-dimensional object detection unit, 24...vehicle search unit, 25...image area comparison unit, 26,26A...vehicle determination unit, A,Ax,Ay...overlapping rate, B...vehicle determination suppression processing, C...determination condition, D...continuous structure degree, PIC...stereo image, PL...left image, PR...right image, PZ...distance image, Robj...three-dimensional object area, Rp...processing target area, Rv...vehicle area, THx,THy...threshold value.

Claims

1. a first area setting unit that sets a first image area by detecting a three-dimensional object based on a distance image generated based on a stereo image; a second region setting unit that sets a second image region by estimating a vehicle degree indicating a degree to which the three-dimensional object is a vehicle based on one of a left image and a right image included in the stereo image; a comparison unit that compares the first image area with the second image area based on a size of the first image area, a size of the second image area, and a relative positional relationship between the first image area and the second image area; a determination unit that determines whether the three-dimensional object is a vehicle based on the vehicle degree and a processing result of the comparison unit; An image processing device comprising:

2. the comparison unit compares the first image area with the second image area by checking whether the first image area and the second image area satisfy a predetermined determination condition; the first image area and the second image area are areas in a first direction and a second direction, The predetermined judgment condition is The second image area is included in the first image area in the first direction; and a value obtained by dividing the width of the second image region in the first direction by the width of the first image region in the first direction is smaller than a predetermined threshold value; Includes The image processing device according to claim 1 .

3. The determination unit performs a vehicle determination suppression process to make it difficult for the three-dimensional object to be determined to be a vehicle when the predetermined determination condition is satisfied. The image processing device according to claim 2 .

4. An estimation unit is further provided for estimating a degree of continuity indicating a degree to which the three-dimensional object is a continuity structure, The determination unit determines whether the three-dimensional object is a vehicle based on the vehicle degree and the continuous structure degree when the predetermined determination condition is satisfied.

4. The image processing device according to claim 2 or 3.

5. An image processing method executed in an image processing device, comprising: setting a first image area by detecting a three-dimensional object based on a distance image generated based on a stereo image; setting a second image area by estimating a vehicle degree indicating a degree to which the three-dimensional object is a vehicle based on one of a left image and a right image included in the stereo image; performing a comparison process for comparing the first image area with the second image area based on a size of the first image area, a size of the second image area, and a relative positional relationship between the first image area and the second image area; determining whether the three-dimensional object is a vehicle based on the vehicle degree and a processing result of the comparison processing; An image processing method comprising:

Images