Image processing device

Abstract

An image processing device according to an embodiment of the present disclosure comprises a processing circuit that can perform filter processing using a prescribed filter pattern for each of a left image and a right image, and can generate a depth image by performing stereo matching processing on the basis of the filtered left and right images. The filter pattern includes a plurality of first filter coefficients which are provided in a first region and which have values of a first polarity, and a plurality of second filter coefficients which are provided in a second region located around the first region and which have values of a second polarity different from the first polarity. The filter coefficient with the largest absolute value among the plurality of first filter coefficients is provided in the center of the first region. The plurality of second filter coefficients include two or more filter coefficients with mutually different values. The filter pattern is a four-fold rotationally symmetric pattern.

Description

Image processing device

[0001] The present disclosure relates to an image processing device that performs a filter process on each of a left image and a right image.

[0002] In vehicles, stereo matching processing is often performed based on left and right images generated by a stereo camera, and driving assistance is performed based on the processing result of the stereo matching processing. Since the images may contain various noises, efforts are made to reduce such noises. For example, Patent Literature 1 discloses a technology for reducing noises such as zipper artifacts contained in images.

[0003] JP 2009-100150 A

[0004] An image processing device according to an embodiment of the present disclosure includes a processing circuit. The processing circuit is capable of performing filter processing on each of a left image and a right image using a predetermined filter pattern, and is capable of generating a distance image by performing stereo matching processing based on the filtered left image and right image. The filter pattern includes a plurality of first filter coefficients provided in a first region and having a first polarity value, and a plurality of second filter coefficients provided in a second region surrounding the first region and having a second polarity value different from the first polarity. A filter coefficient having the largest absolute value among the plurality of first filter coefficients is provided in the center of the first region. The plurality of second filter coefficients include two or more filter coefficients having different values ​​from each other. The filter pattern is a four-fold rotationally symmetric pattern.

[0005] The accompanying drawings are included to provide a further understanding of the disclosure, and are incorporated in and constitute a part of this specification. The drawings illustrate one embodiment and, together with the description, serve to explain the principles of the disclosure.

[0006] FIG. 1 is an explanatory diagram illustrating an example configuration of a vehicle equipped with a driving assistance device according to an embodiment of the present disclosure. FIG. 2 is a block diagram illustrating an example configuration of the driving assistance device illustrated in FIG. 1. FIG. 3 is an explanatory diagram illustrating an example pixel array in an image sensor of a stereo camera illustrated in FIG. 2. FIG. 4 is an explanatory diagram illustrating examples of left and right images illustrated in FIG. 2. FIG. 5 is an explanatory diagram illustrating an example of zipper noise. FIG. 6 is an explanatory diagram illustrating an example configuration of a filter pattern used by a filter processing unit illustrated in FIG. 2. FIG. 7 is an explanatory diagram illustrating an example of creating the filter pattern illustrated in FIG. 6. FIG. 8 is another explanatory diagram illustrating an example of creating the filter pattern illustrated in FIG. 6. FIG. 9 is another explanatory diagram illustrating an example of creating the filter pattern illustrated in FIG. 6. FIG. 10 is an explanatory diagram illustrating an example of a processing target image on which filtering is performed by the filter processing unit illustrated in FIG. 2. FIG. 11 is an explanatory diagram illustrating an example of filtering in the filter processing unit illustrated in FIG. 2. FIG. 12 is an explanatory diagram illustrating an example of filtering according to a reference example. FIG. 13 is an explanatory diagram illustrating an example of filtering according to another reference example. Fig. 14 is an explanatory diagram showing an example of the traveling speed of a preceding vehicle estimated by the driving assistance device shown in Fig. 2. Fig. 15 is an explanatory diagram showing an example of the traveling speed of a preceding vehicle estimated by a driving assistance device according to another reference example.

[0007] When stereo matching is performed based on the left and right images, noise contained in the left and right images can reduce the accuracy of the stereo matching. Therefore, it is expected that the reduction in accuracy of the stereo matching can be suppressed.

[0008] It is desirable to provide an image processing device that can suppress a decrease in accuracy of stereo matching processing.

[0009] Some exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that the following description illustrates one specific example of the present disclosure and should not be construed as limiting the present disclosure. For example, each element, including numerical values, shapes, materials, parts, the position of each part, and the connection method of each part, is merely an example and should not be construed as limiting the present disclosure. Furthermore, in the following exemplary embodiments, components not described in independent claims based on the highest concept of the present disclosure are optional and may be provided as needed. The drawings are schematic and are not intended to be drawn to scale. Throughout this specification and the drawings, components having substantially the same function and configuration are designated by the same reference numerals, and redundant description will be omitted. Furthermore, components not directly related to one embodiment of the present disclosure are not shown in the drawings.

[0010] 1 and 2 show an example of the configuration of a driving assistance device 1 equipped with an image processing device according to an embodiment. The driving assistance device 1 is mounted on a vehicle 9 and is configured to assist a driver in driving the vehicle 9. The driving assistance device 1 includes a stereo camera 10 and a processing device 20.

[0011] The stereo camera 10 is configured to capture images ahead of the vehicle 9 and generate data of a set of images having parallax from each other. The stereo camera 10 includes a left camera 11L, a right camera 11R, and a demosaic processing unit 12. Each of the left camera 11L and the right camera 11R includes a lens and an image sensor.

[0012] FIG. 3 shows an example of a pixel array in the image sensor of the stereo camera 10. This image sensor has a plurality of pixels Pix arranged in parallel. The plurality of pixels Pix includes a pixel Pix capable of detecting red (R) light, a pixel Pix capable of detecting green (G) light, and a pixel Pix capable of detecting blue (B) light. The plurality of pixels Pix are arranged in a unit U, each of which has four pixels Pix (=2×2) arranged in two rows and two columns. In this example, in the unit U, a pixel Pix capable of detecting red (R) light is arranged in the lower left, pixels Pix capable of detecting green (G) light are arranged in the upper left and lower right, and a pixel Pix capable of detecting blue (B) light is arranged in the upper right. This pixel array is also called a Bayer array.

[0013] 1, the stereo camera 10 is disposed inside the vehicle 9 near the upper portion of the windshield of the vehicle 9. The left camera 11L and the right camera 11R of the stereo camera 10 are disposed a predetermined distance apart in the width direction of the vehicle 9. The left camera 11L generates a left image, and the right camera 11R generates a right image.

[0014] The demosaic processing unit 12 is configured to perform demosaic processing on each of the left image supplied from the left camera 11L and the right image supplied from the right camera 11R. In an image sensor in which pixels Pix are arranged in a Bayer array, pixels Pix capable of detecting red (R) light are provided at a ratio of one pixel Pix per four pixels Pix, for example, as shown in FIG. 3 . Therefore, the demosaic processing unit 12 generates a red image by calculating, by interpolation, pixel values ​​at positions where no pixels Pix capable of detecting red (R) light are provided, based on pixel values ​​at the pixels Pix capable of detecting red (R) light. Similarly, pixels Pix capable of detecting green (G) light are provided at a ratio of one pixel Pix per two pixels Pix, for example, as shown in FIG. 3 . Therefore, the demosaic processing unit 12 generates a green image by calculating, by interpolation, pixel values ​​at positions where no pixels Pix capable of detecting green (G) light are provided, based on pixel values ​​at the pixels Pix capable of detecting green (G) light. 3, pixels Pix capable of detecting blue (B) light are provided at a ratio of one pixel Pix per four pixels Pix. The demosaic processing unit 12 generates a blue image by calculating pixel values ​​at positions where no pixels Pix capable of detecting blue (B) light are provided, based on pixel values ​​of the pixels Pix capable of detecting blue (B) light, through interpolation. In this manner, the demosaic processing unit 12 performs demosaic processing on the left image supplied from the left camera 11L to generate a left image PL including a red image, a green image, and a blue image, and performs demosaic processing on the right image supplied from the right camera 11R to generate a right image PR including a red image, a green image, and a blue image.

[0015] In this way, the stereo camera 10 generates a left image PL and a right image PR, which together form a stereo image PIC.

[0016] 4 shows an example of a left image PL and a right image PR constituting a stereo image PIC. In this example, another vehicle (a preceding vehicle 8) is traveling ahead of the vehicle 9 on the road on which the vehicle 9 is traveling. The left camera 11L captures an image of the preceding vehicle 8 to generate the left image PL, and the right camera 11R captures an image of the preceding vehicle 8 to generate the right image PR. The left camera 11L and the right camera 11R are arranged at a predetermined distance apart in the width direction of the vehicle 9, and therefore the left image PL and the right image PR have a parallax corresponding to the difference in their positions.

[0017] The stereo camera 10 performs an imaging operation at a predetermined frame rate (e.g., 60 fps) to generate a series of stereo images PIC, and supplies image data of the generated series of stereo images PIC to the processing device 20.

[0018] The processing device 20 is configured to perform processing based on the left image PL and the right image PR to control the operation of the driving assistance device 1. The processing device 20 is configured using, for example, one or more processors, one or more memories, etc., and is configured to perform processing by executing a program. The processing device 20 has a grayscale image generation unit 21, a filter processing unit 22, a parallax image generation unit 23, a distance image generation unit 24, and a driving assistance processing unit 25.

[0019] The grayscale image generation unit 21 is configured to generate a grayscale image relating to the left image PL based on the red image, green image, and blue image contained in the left image PL, and to generate a grayscale image relating to the right image PR based on the red image, green image, and blue image contained in the right image PR.

[0020] The filter processing unit 22 is configured to perform filter processing on each of the grayscale image corresponding to the left image PL and the grayscale image corresponding to the right image PR. As will be described later, the parallax image generation unit 23 of the processing device 20 performs stereo matching processing based on the two grayscale images corresponding to the left image PL and the right image PR that have been filtered by the filter processing unit 22. In order to improve the accuracy of the stereo matching processing, the filter processing unit 22 performs filter processing to emphasize features of the subject in the two grayscale images. However, if these two grayscale images contain, for example, zipper noise as shown below, and the zipper noise is emphasized, the accuracy of the stereo matching processing may be reduced.

[0021] FIG. 5 shows an example of zipper noise. In this example, the noise pattern of the zipper noise has a pattern in which light and dark shading is repeated in a certain direction (the horizontal direction in this example), for example, in units of one pixel. In this example, the noise pattern includes two lines of patterns, but this is not limited thereto. Alternatively, the noise pattern may include a one-line pattern, or a three-line or more pattern. Also, in this example, the noise pattern repeats light and dark shading in units of one pixel, but this is not limited thereto. Alternatively, the noise pattern may repeat light and dark shading in units of a small number of pixels, such as two pixels. Also, in this example, the noise pattern is a one-dimensional pattern extending in a certain direction, but it may also be a two-dimensional pattern extending in both the vertical and horizontal directions.

[0022] If these two grayscale images contain such zipper noise and the zipper noise is emphasized, a mismatch may occur in the stereo matching process, and the accuracy of the stereo matching process may decrease. Therefore, the filter processing unit 22 performs a filter process on each of the grayscale image related to the left image PL and the grayscale image related to the right image PR so as to emphasize the features of the subject and not emphasize the zipper noise. Specifically, the filter processing unit 22 performs this filter process by performing a convolution operation using the filter pattern PAT shown below.

[0023] FIG. 6 shows an example of a filter pattern PAT. This filter pattern PAT has 25 (5 × 5) filter coefficients arranged in 5 rows and 5 columns. This filter pattern PAT is divided into regions R1 and R2. Region R1 is located near the center of the filter pattern PAT and includes nine (3 × 3) filter coefficients arranged in 3 rows and 3 columns. In FIG. 6, region R1 is indicated by shading. Region R2 is located around region R1, surrounding region R1, and includes 16 filter coefficients. In this example, the polarities of the nine filter coefficients arranged in region R1 are positive, and the polarities of the 16 filter coefficients arranged in region R2 are negative. In region R1, the filter coefficient with the largest absolute value is located in the center of region R1. The absolute values ​​of the nine filter coefficients in region R1 decrease as the region moves away from the center of region R1. The 16 filter coefficients in region R2 include eight filter coefficients with a value of "-2" and eight filter coefficients with a value of "-1". This filter pattern PAT is a four-fold rotationally symmetric pattern. In other words, the filter pattern PAT remains the same even when rotated 90 degrees.

[0024] The filter processing unit 22 performs a filter process by performing a convolution operation on each of the grayscale image related to the left image PL and the grayscale image related to the right image PR using such a filter pattern PAT.

[0025] The parallax image generation unit 23 is configured to generate a parallax image by performing stereo matching processing based on two grayscale images of the left image PL and the right image PR that have been filtered by the filtering processing unit 22. Specifically, the parallax image generation unit 23 performs stereo matching processing based on these two grayscale images to identify corresponding points including two corresponding image points (left image points and right image points). The left image points are image points in the grayscale image of the left image PL that has been filtered, and the right image points are image points in the grayscale image of the right image PR that has been filtered. The parallax image generation unit 23 then generates a parallax image by calculating a parallax value based on the difference between the positions of the left image points and the right image points. The pixel values ​​in this parallax image are the parallax values.

[0026] The distance image generator 24 is configured to generate a distance image by converting pixel values ​​included in the parallax image from parallax values ​​to distance values ​​based on the parallax image. This distance value indicates the distance from the stereo camera 10 to the subject.

[0027] The driving assistance processing unit 25 is configured to provide driving assistance for the vehicle 9. The driving assistance processing unit 25 recognizes a subject based on, for example, the left image PL and the right image PR transmitted from the stereo camera 10. The driving assistance processing unit 25 also calculates a speed difference between the vehicle 9 and a leading vehicle based on, for example, a distance image, and estimates the traveling speed of the leading vehicle based on this speed difference and the traveling speed of the vehicle 9. The driving assistance processing unit 25 controls the operation of the vehicle 9, for example, to notify the driver of these processing results. The driving assistance processing unit 25 also controls the operation of the vehicle 9 based on, for example, the distance image, so that the vehicle 9 travels following the leading vehicle.

[0028] Here, the driving assistance device 1 corresponds to a specific example of an "image processing device" in an embodiment of the present disclosure. The processing device 20 corresponds to a specific example of a "processing circuit" in an embodiment of the present disclosure. The grayscale image of the left image PL corresponds to a specific example of a "left image" in an embodiment of the present disclosure. The grayscale image of the right image PR corresponds to a specific example of a "right image" in an embodiment of the present disclosure. The filter pattern PAT corresponds to a specific example of a "filter pattern" in an embodiment of the present disclosure. The region R1 corresponds to a specific example of a "first region" in an embodiment of the present disclosure. The region R2 corresponds to a specific example of a "second region" in an embodiment of the present disclosure.

[0029] [Operation and Function] Next, the operation and function of the driving assistance device 1 of this embodiment will be described.

[0030] (Overview of Overall Operation) First, the operation of the driving assistance device 1 will be described with reference to FIG. 2 . The stereo camera 10 captures images of the area ahead of the vehicle 9 to generate a stereo image PIC including a left image PL and a right image PR. The grayscale image generation unit 21 generates a grayscale image of the left image PL based on the red, green, and blue images included in the left image PL, and generates a grayscale image of the right image PR based on the red, green, and blue images included in the right image PR. The filter processing unit 22 performs filtering by performing a convolution operation on each of the grayscale images of the left image PL and the right image PR using a filter pattern PAT. The parallax image generation unit 23 generates a parallax image by performing stereo matching on the two grayscale images of the left image PL and the right image PR that have been subjected to the filtering process. The distance image generation unit 24 generates a distance image by converting pixel values ​​included in the parallax image from parallax values ​​to distance values ​​based on the parallax image. The driving assistance processing unit 25 provides driving assistance for the vehicle 9 .

[0031] (Detailed Operation) The filter processing unit 22 performs filter processing on each of the grayscale image corresponding to the left image PL and the grayscale image corresponding to the right image PR so as to emphasize the features of the subject while not emphasizing zipper noise. Specifically, the filter processing unit 22 performs this filter processing by performing a convolution operation using the filter pattern PAT shown in Fig. 6. A method for creating the filter pattern PAT will be described in detail below.

[0032] 7 to 9 show an example of a method for creating a filter pattern PAT. In this example, the filter pattern PAT is created using a Sobel filter and a Gaussian filter. The Sobel filter is a sharpening filter used to emphasize the features of the object in the image. The Gaussian filter is a smoothing filter used to prevent zipper noise from being emphasized.

[0033] First, two Sobel filter patterns P1 and P2 are prepared as shown in Fig. 7. Each of these filter patterns P1 and P2 has nine (=3 x 3) filter coefficients arranged in three rows and three columns.

[0034] In filter pattern P1, the filter coefficients in the left column have negative polarity, the filter coefficients in the center column have zero polarity, and the filter coefficients in the right column have positive polarity. The absolute values ​​of the three filter coefficients in the left column and the three filter coefficients in the right column are equal to each other. Filter pattern P1 is symmetrical in the vertical direction. Filter processing using such filter pattern P1 can emphasize changes in pixel values ​​in the horizontal direction.

[0035] In filter pattern P2, the filter coefficients in the top row have negative polarity, the filter coefficients in the middle row have zero polarity, and the filter coefficients in the bottom row have positive polarity. The absolute values ​​of the three filter coefficients in the top row and the absolute values ​​of the three filter coefficients in the bottom row are equal to each other. Filter pattern P2 is symmetrical in the horizontal direction. Filter processing using such filter pattern P2 can emphasize changes in pixel values ​​in the vertical direction.

[0036] Next, filter pattern P1 is modified to create filter pattern P3. This filter pattern P3 has 25 (5 × 5) filter coefficients arranged in 5 rows and 5 columns. In filter pattern P3, the polarity of the filter coefficient in the leftmost column is negative, the polarity of the filter coefficient in the second column from the left is zero, the polarity of the filter coefficient in the center column is positive, the polarity of the filter coefficient in the second column from the right is zero, and the polarity of the filter coefficient in the rightmost column is negative. The five filter coefficients in the leftmost column and the five filter coefficients in the rightmost column are equal to each other. Filter pattern P3 is symmetrical in both the vertical and horizontal directions. Filter processing using such filter pattern P3 can emphasize changes in pixel values ​​in the horizontal direction.

[0037] Similarly, filter pattern P2 is modified to create filter pattern P4. Filter pattern P4 has 25 (5 × 5) filter coefficients arranged in 5 rows and 5 columns. In filter pattern P4, the polarity of the filter coefficients in the top row is negative, the polarity of the filter coefficients in the second row from the top is zero, the polarity of the filter coefficients in the middle row is positive, the polarity of the filter coefficients in the second row from the bottom is zero, and the polarity of the filter coefficients in the bottom row is negative. The five filter coefficients in the top row and the five filter coefficients in the bottom row are equal to each other. Filter pattern P4 is symmetrical in both the vertical and horizontal directions. Filter pattern P4 is the same as filter pattern P3 rotated 90 degrees. Filter processing using such filter pattern P4 can emphasize changes in pixel values ​​in the vertical direction.

[0038] As will be described later, filter pattern PAT is ultimately generated by adding filter pattern P6, which is based on filter patterns P3 and P4, to filter pattern P7, which has a pattern of 3 rows and 3 columns. Considering that filter pattern P7 is a pattern of 3 rows and 3 columns, filter patterns P3 and P4 are made into patterns of 5 rows and 5 columns, which is larger than filter pattern P7.

[0039] Next, filter pattern P5 is created by adding filter pattern P3 and filter pattern P4 together. Since filter pattern P4 is the same as filter pattern P3 rotated 90 degrees, filter pattern P5, created by adding these patterns together, is a four-fold rotationally symmetric pattern. In other words, filter pattern P5 remains the same even when rotated 90 degrees. In filter pattern P5, the values ​​of nine filter coefficients (=3×3) arranged in three rows and three columns near the center are positive polarity values ​​or zero. In filter pattern P5, the values ​​of 16 filter coefficients arranged in positions surrounding these nine filter coefficients are negative polarity values ​​or zero.

[0040] As shown by the dashed-dotted lines in FIG. 8 , the center filter coefficient of the leftmost column, the center filter coefficient of the rightmost column, the center filter coefficient of the top row, and the center filter coefficient of the bottom row in this filter pattern P5 are zero. In this case, it is difficult to emphasize changes in pixel values ​​in the horizontal direction, and similarly it is difficult to emphasize changes in pixel values ​​in the vertical direction. Therefore, filter pattern P6 is created by adjusting the values ​​of these four filter coefficients in filter pattern P5. In this example, the center filter coefficient of the leftmost column and the center filter coefficient of the rightmost column in filter pattern P3, and the center filter coefficient of the top row and the center filter coefficient of the bottom row in filter pattern P4, shown by the dashed-dotted lines in FIG. 7 , are used to set these four filter coefficients in filter pattern P6 to "-2." Filter processing using such filter pattern P6 can emphasize changes in pixel values ​​in the vertical and horizontal directions.

[0041] Next, a filter pattern P7 of a Gaussian filter is prepared, as shown in FIG. 9 . This filter pattern P7 has nine filter coefficients (=3×3) arranged in three rows and three columns. These nine filter coefficients have positive polarity. Of the nine filter coefficients, the filter coefficient arranged in the center is the largest. The nine filter coefficients further away from the center become smaller. The filter pattern P7 is a four-fold rotationally symmetric pattern. In other words, the filter pattern P7 remains the same even when rotated 90 degrees.

[0042] The Gaussian filter is used to prevent the zipper noise from being emphasized. The noise pattern of the zipper noise repeats light and dark on a pixel-by-pixel basis, as shown in Figure 5. Therefore, in this example, a small pattern such as 3 rows and 3 columns is sufficient.

[0043] Then, as shown in Fig. 9, the filter pattern PAT shown in Fig. 6 is created by adding together the filter patterns P6 and P7. Specifically, the filter pattern PAT is created by adding the nine filter coefficients (=3×3) of the filter pattern P7 to the nine filter coefficients (=3×3) of the filter pattern P6 arranged in a 3-row, 3-column pattern near the center. This filter pattern PAT combines the features of the filter patterns P6 and P7. Therefore, filtering using such a filter pattern PAT can emphasize the features of the subject while de-emphasizing zipper noise.

[0044] (Example of Filtering Process) Next, an example of filtering process using the filter pattern PAT will be described.

[0045] 10 shows an example of a processing target image PA to be subjected to filtering. The processing target image PA corresponds to a grayscale image supplied to the filtering processing unit 22. This processing target image PA is divided by a boundary line B1 into a left half with large pixel values ​​and a right half with small pixel values. In the left half, partial images W1 each containing four pixel values ​​are repeatedly arranged, and in the right half, partial images W2 each containing four pixel values ​​are repeatedly arranged.

[0046] In partial image W1 in the left half, the upper left pixel value is "176," the lower left pixel value is "160," the upper right pixel value is "160," and the lower right pixel value is "144." In this partial image W1, the difference between two vertically aligned pixel values ​​is "16," and the difference between two horizontally aligned pixel values ​​is "16." In partial image W2 in the right half, the upper left pixel value is "112," the lower left pixel value is "96," the upper right pixel value is "96," and the lower right pixel value is "80." In this partial image W2, the difference between two vertically aligned pixel values ​​is "16," and the difference between two horizontally aligned pixel values ​​is "16." By repeatedly arranging partial image W1 in the left half and partial image W2 in the right half, two-dimensional zipper noise is formed. In this example, the difference between the two pixel values ​​on either side of boundary line B1 is "48."

[0047] 11 shows an example of filter processing using a filter pattern PAT, where (A) shows the filter pattern PAT and (B) shows an image generated by the filter processing. Note that the scale factor of this filter pattern PAT is "8." The scale factor of the filter pattern PAT is the absolute value of the sum of the 25 filter coefficients in the filter pattern PAT. In the filter processing, the scale of pixel values ​​is adjusted based on this scale factor.

[0048] As shown in FIG. 11B , the image generated by the filter processing is divided into a left half with high pixel values ​​and a right half with low pixel values, with boundary line B2 in between, similar to the processing target image PA. In this example, the difference between the two pixel values ​​on the left and right sides of boundary line B2 is "144," which is larger than the difference between the two pixel values ​​on the left and right sides of boundary line B1 in the processing target image PA shown in FIG. 10 . As such, near boundary line B2 in the image generated by the filter processing, pixel values ​​change abruptly in the horizontal direction. In other words, this filter processing emphasizes edges. Therefore, this filter processing is expected to emphasize the features of the subject.

[0049] For example, in partial image W3 in the left half, the upper left pixel value is "144," the lower left pixel value is "160," the upper right pixel value is "160," and the lower right pixel value is "176." In this partial image W3, the difference between two vertically aligned pixel values ​​is "16," and the difference between two horizontally aligned pixel values ​​is "16." These values ​​are the same as the difference between two vertically aligned pixel values ​​("16") and the difference between two horizontally aligned pixel values ​​("16") in partial image W1 of the processing target image PA shown in FIG. 10. Similarly, in partial image W4 in the right half, the upper left pixel value is "80," the lower left pixel value is "96," the upper right pixel value is "96," and the lower right pixel value is "112." In this partial image W4, the difference between two vertically aligned pixel values ​​is "16," and the difference between two horizontally aligned pixel values ​​is "16." These values ​​are the same as the difference "16" between two pixel values ​​aligned vertically and the difference "16" between two pixel values ​​aligned horizontally in partial image W2 of the processing target image PA shown in Fig. 10. Therefore, zipper noise is not emphasized in this filtering process.

[0050] In this way, the filter processing using the filter pattern PAT can emphasize the features of the subject while preventing zipper noise from being emphasized.

[0051] Next, as reference examples, an example of filter processing using a Gaussian filter, which is a smoothing filter, and an example of filter processing using an unsharp filter, which is a sharpening filter, will be described.

[0052] Reference Example E1 A case will be described in which a filter process is performed on the processing target image PA shown in FIG. 10 using a Gaussian filter, which is a smoothing filter.

[0053] 12 shows an example of filtering using a Gaussian filter, where (A) shows the filter pattern and (B) shows the image generated by the filtering. The scale factor of this filter pattern is 16.

[0054] The image generated by the filter processing is divided, like the processing target image PA, into a left half with large pixel values ​​and a right half with small pixel values, separated by a boundary line B3. For example, in partial image W5 in the left half, all four pixel values ​​are "160." Similarly, in partial image W6 in the right half, all four pixel values ​​are "96." In other words, in this example, a Gaussian filter, which is a smoothing filter, is used, so the pixel values ​​are smoothed. Therefore, zipper noise is reduced in this filter processing.

[0055] However, in this example, the difference between the two pixel values ​​on either side of boundary line B3 is "32," which is smaller than the difference between the two pixel values ​​on either side of boundary line B1 in the processing target image PA shown in FIG. 10, which is "48." As shown in FIG. 12, pixel values ​​change gradually in the horizontal direction near boundary line B3. In other words, this filtering process produces a gentler edge. In this case, this filtering process does not emphasize the features of the subject, which may result in a decrease in the accuracy of the stereo matching process.

[0056] Reference Example E2 A case will be described in which filtering is performed on the processing target image PA shown in FIG. 10 using an unsharp filter, which is a sharpening filter.

[0057] 13 shows an example of filter processing using an unsharp filter, where (A) shows the filter pattern and (B) shows the image generated by the filter processing. Note that the scale factor of this filter pattern is "16."

[0058] The image generated by the filter process is divided into a left half with large pixel values ​​and a right half with small pixel values, as in the processing target image PA, with boundary line B4 in between. In this example, the difference between the two pixel values ​​on the left and right sides of boundary line B4 is "64," which is larger than the difference between the two pixel values ​​on the left and right sides of boundary line B1 in the processing target image PA, which is "48." In other words, this filter process emphasizes edges. Therefore, this filter process is expected to emphasize the features of the subject.

[0059] However, in partial image W7 in the left half, the upper left pixel value is "128," the lower left pixel value is "160," the upper right pixel value is "160," and the lower right pixel value is "192." In this partial image W7, the difference between two vertically aligned pixel values ​​is "32," and the difference between two horizontally aligned pixel values ​​is "32." These values ​​are twice the difference between two vertically aligned pixel values ​​("16") and two horizontally aligned pixel values ​​("16") in partial image W1 of the processing target image PA shown in FIG. 10. Similarly, in partial image W8 in the right half, the upper left pixel value is "64," the lower left pixel value is "96," the upper right pixel value is "96," and the lower right pixel value is "128." In this partial image W8, the difference between two vertically aligned pixel values ​​is "32," and the difference between two horizontally aligned pixel values ​​is "32." These values ​​are twice the difference "16" between two vertically aligned pixel values ​​and the difference "16" between two horizontally aligned pixel values ​​in partial image W2 of the processing target image PA shown in Figure 10. Thus, zipper noise is also emphasized in this filtering process. If zipper noise is emphasized in this way, the accuracy of the stereo matching process may decrease.

[0060] On the other hand, in the driving assistance device 1 of this embodiment, the filter pattern PAT shown in FIG. 6 is used to perform filtering, which enables edges to be emphasized while zipper noise is not emphasized. This allows the driving assistance device 1 to emphasize the features of the subject, thereby improving the accuracy of the stereo matching process, and also prevents zipper noise from being emphasized, thereby preventing a decrease in the accuracy of the stereo matching process due to zipper noise. As a result, the driving assistance device 1 can improve the accuracy of the parallax image and the distance image.

[0061] (Estimation accuracy of driving speed) For example, the driving assistance processing unit 25 calculates the speed difference between the vehicle 9 and the preceding vehicle based on the distance image, and estimates the driving speed of the preceding vehicle based on this speed difference and the driving speed of the vehicle 9.

[0062] 14 shows an example of the traveling speed of the preceding vehicle estimated by the driving assistance processing unit 25. The horizontal axis indicates the frame number, and the vertical axis indicates the estimated traveling speed of the preceding vehicle. In this example, the estimated traveling speed increases relatively smoothly over time.

[0063] 15 shows an example of an estimated value of the traveling speed of a preceding vehicle according to a reference example. In this example, filtering using a sharpening filter is performed instead of filtering using the filter pattern PAT. In this case, fluctuations occur in the estimated value of the traveling speed.

[0064] When filter processing is performed using the filter pattern PAT (FIG. 14), fluctuations in the estimated value of the traveling speed can be suppressed compared to when filter processing is performed using a sharpening filter (FIG. 15). In other words, by performing filter processing using the filter pattern PAT, the driving assistance device 1 can suppress a decrease in accuracy of the stereo matching processing caused by zipper noise, and can suppress a decrease in accuracy of the distance image. As a result, the accuracy of estimating the traveling speed can be improved.

[0065] As described above, the driving assistance device 1 is capable of performing filter processing on each of the left image (grayscale image related to the left image PL) and the right image (grayscale image related to the right image PR) using a predetermined filter pattern PAT. The driving assistance device 1 is equipped with a processing circuit (processing device 20) capable of generating a distance image by performing stereo matching processing based on the filtered left and right images. The filter pattern PAT includes a plurality of first filter coefficients provided in a first region (region R1) and having a first polarity value, and a plurality of second filter coefficients provided in a second region (region R2) surrounding the first region (region R1) and having a second polarity value different from the first polarity. The filter coefficient with the largest absolute value among the plurality of first filter coefficients is provided in the center of the first region. The plurality of second filter coefficients include two or more filter coefficients having different values. The filter pattern is a four-fold rotationally symmetric pattern. As a result, the driving assistance device 1 can improve the accuracy of the stereo matching process by emphasizing the features of the subject in the left and right images, and can suppress a decrease in the accuracy of the stereo matching process caused by zipper noise by not emphasizing zipper noise.

[0066] In the driving assistance device 1, the processing circuit (processing device 20) is configured to perform filter processing using a single filter pattern PAT. This allows the processing device 20 to reduce the amount of calculations, making processing possible even in devices with limited processing resources, such as embedded devices. For example, performing filter processing using a sharpening filter pattern and filter processing using a smoothing filter pattern separately would result in a large amount of processing. In this case, it would be difficult to perform the processing in devices with limited processing resources. In the driving assistance device 1, filter processing is performed using a filter pattern PAT that includes the characteristics of both the sharpening filter and the smoothing filter. By performing filter processing using a single filter pattern PAT in this way, only one filter processing is required, thereby reducing the processing amount. Therefore, processing is possible in devices with limited processing resources, for example.

[0067] In the driving assistance device 1, the absolute values ​​of the plurality of first filter coefficients in the first region (region R1) are set to be smaller as the distance from the center of the first region (region R1) increases. This allows the filter pattern PAT to include the characteristics of a smoothing filter, making it possible to prevent zipper noise from being emphasized. As a result, it is possible to prevent a decrease in the accuracy of the stereo matching process due to zipper noise.

[0068] In the driving assistance device 1, at least one of the left image (grayscale image related to the left image PL) and the right image (grayscale image related to the right image PR) includes a noise pattern that repeats light and dark shades in units of one or more pixel values ​​in a predetermined direction. In this way, even when the grayscale image includes such a noise pattern, the driving assistance device 1 can prevent the noise of this noise pattern from being emphasized, thereby suppressing a decrease in the accuracy of the stereo matching process.

[0069] In the driving assistance device 1, the left image (grayscale image related to the left image PL) and the right image (grayscale image related to the right image PR) can be generated by demosaic processing. In demosaic processing, pixel values ​​are calculated by interpolation, which can cause zipper noise. Even in such cases, the driving assistance device 1 can prevent the zipper noise from being emphasized, thereby suppressing a decrease in the accuracy of the stereo matching processing due to the zipper noise.

[0070] [Effects] As described above, this embodiment includes a processing circuit capable of performing filter processing on each of the left and right images using a predetermined filter pattern PAT and generating a distance image by performing stereo matching processing based on the filtered left and right images. The filter pattern includes a plurality of first filter coefficients provided in a first region and having a first polarity value, and a plurality of second filter coefficients provided in a second region surrounding the first region and having a second polarity value different from the first polarity. The filter coefficient with the largest absolute value among the plurality of first filter coefficients is provided in the center of the first region. The plurality of second filter coefficients include two or more filter coefficients with different values. The filter pattern is a four-fold rotationally symmetric pattern. This makes it possible to suppress a decrease in the accuracy of the stereo matching processing.

[0071] In this embodiment, the processing circuit is capable of performing filter processing using a single filter pattern, so that processing can be performed in a device that does not have abundant processing resources.

[0072] In this embodiment, the absolute values ​​of the multiple first filter coefficients in the first region are each set to be smaller the further away from the center of the first region, so that the decrease in accuracy of the stereo matching process can be suppressed.

[0073] In the driving assistance device 1, at least one of the left image and the right image includes a noise pattern that repeats light and dark shades in units of one or more pixel values ​​in a predetermined direction, thereby suppressing a decrease in the accuracy of the stereo matching process.

[0074] In the driving assistance device 1, the left and right images can be generated by demosaicing. In this case, zipper noise may occur in the left and right images, but even in such a case, it is possible to suppress a decrease in the accuracy of the stereo matching process.

[0075] Although an example of an embodiment of the present disclosure has been described above with reference to the accompanying drawings, the present disclosure is by no means limited to the above embodiment. Those skilled in the art will understand that various modifications and variations can be made without departing from the scope defined by the appended claims. The present disclosure is intended to encompass such modifications and variations to the extent that they fall within the scope of the appended claims and their equivalents.

[0076] For example, in the above embodiment, 25 filter coefficients are set as shown in FIG. 6, but the present invention is not limited to this, and the 25 filter coefficients can be changed as appropriate.

[0077] For example, in the above embodiment, as shown in Fig. 6, the filter has 25 filter coefficients (=5x5) arranged in 5 rows and 5 columns, but this is not limited to this, and instead, for example, the filter may have 49 filter coefficients (=7x7) arranged in 7 rows and 7 columns. In this case, the region R1 may include, for example, 9 filter coefficients (=3x3) arranged in 3 rows and 3 columns, or 25 filter coefficients (=5x5) arranged in 5 rows and 5 columns.

[0078] The effects described in this specification are merely examples, and the effects of the present disclosure are not limited to the effects described in this specification. Therefore, other effects may be obtained with respect to the present disclosure.

[0079] Furthermore, the present disclosure may take the following aspects.

[0080] (1) An image processing device including a processing circuit capable of performing filtering on each of a left image and a right image using a predetermined filter pattern, and capable of generating a distance image by performing stereo matching processing based on the left image and the right image that have been subjected to the filtering processing, wherein the filter pattern includes a plurality of first filter coefficients provided in a first region and having a first polarity value, and a plurality of second filter coefficients provided in a second region disposed around the first region and having a second polarity value different from the first polarity, wherein a filter coefficient having a largest absolute value among the plurality of first filter coefficients is provided in a center of the first region, and the plurality of second filter coefficients include two or more filter coefficients having different values, and the filter pattern is a pattern of four-fold rotational symmetry. (2) The image processing device described in (1), wherein the processing circuit is capable of performing the filtering using a single filter pattern. (3) The image processing device described in (1) or (2), wherein the absolute values ​​of the plurality of first filter coefficients in the first region decrease as the distance from the center of the first region increases. (4) The image processing device according to any one of (1) to (3), wherein at least one of the left image and the right image includes a noise pattern in which light and dark shading is repeated in units of one or more pixel values ​​in a predetermined direction. (5) The image processing device according to any one of (1) to (4), wherein the left image and the right image can be generated by demosaic processing.

[0081] The processing device 20 shown in FIG. 2 can be implemented by circuitry including at least one semiconductor integrated circuit, such as at least one processor (e.g., a central processing unit (CPU)), at least one application-specific integrated circuit (ASIC), and / or at least one field-programmable gate array (FPGA). The at least one processor can be configured to perform all or a portion of the various functions of the processing device 20 shown in FIG. 2 by reading instructions from at least one non-transitory, tangible computer-readable medium. Such media can take various forms, including, but not limited to, various magnetic media such as hard disks, various optical media such as CDs or DVDs, and various semiconductor memories (i.e., semiconductor circuits) such as volatile or non-volatile memories. Volatile memories can include DRAM and SRAM. Non-volatile memories can include ROM and NVRAM. An ASIC is an integrated circuit (IC) specialized to perform all or a portion of the various functions of the processing device 20 shown in FIG. 2. An FPGA is an integrated circuit designed to be configurable after manufacture to perform all or a portion of the various functions of the processing device 20 shown in FIG. 2.

Claims

1. An image processing device comprising a processing circuit capable of performing filter processing on each of a left image and a right image using a predetermined filter pattern, and capable of generating a distance image by performing stereo matching processing based on the left image and the right image that have been subjected to the filter processing, wherein the filter pattern includes a plurality of first filter coefficients arranged in a first region and having a first polarity value, and a plurality of second filter coefficients arranged in a second region surrounding the first region and having a second polarity value different from the first polarity, wherein the filter coefficient having the largest absolute value among the plurality of first filter coefficients is arranged in the center of the first region, and the plurality of second filter coefficients include two or more filter coefficients whose values ​​are different from each other, and the filter pattern is a pattern with four-fold rotational symmetry.

2. The image processing device according to claim 1, wherein the processing circuit is capable of performing the filtering process using a single filter pattern.

3. The image processing device according to claim 1, wherein the absolute values ​​of the plurality of first filter coefficients in the first region become smaller as the distance from the center of the first region increases.

4. The image processing device according to claim 1, wherein at least one of the left image and the right image includes a noise pattern in which light and dark shading is repeated in units of one or more pixel values ​​in a predetermined direction.

5. The image processing device according to claim 1, wherein the left image and the right image can be generated by demosaic processing.

Images