Image processing device

Abstract

The present invention provides an image processing device capable of appropriately setting image regions. The image processing device, in one embodiment of the present invention, includes a representative distance calculation unit that generates multiple representative distance values based on a distance image generated from a stereo image, and a merging unit that performs a merging process to merge the set first and second image regions, and determines whether a virtual merging region of the first and second image regions includes an image of a corner portion of another vehicle, avoiding the merging process if the virtual merging region includes an image of a corner portion. The merging unit generates a first approximate straight line based on multiple first representative distance values corresponding to the first image region and calculates a first degree of deviation. It generates a second approximate straight line based on multiple second representative distance values corresponding to the second image region and calculates a second degree of deviation, and performs a determination process based on the first and second degrees of deviation.

Description

Technical Field

[0001] The present invention relates to an image processing device for performing image processing. Background Art

[0002] In an image processing device, an image region is set in a frame image, and various processing is performed based on the image in the image region. For example, Patent Document 1 discloses a feature extraction device that uses clustering processing to segment a three-dimensional dot matrix, which is a collection of three-dimensional points representing positions on the surface of an object, to generate dot matrix clusters, and then extracts feature quantities based on the dot matrix clusters.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-3527 Summary of the Invention

[0006] Technical issues

[0007] It is desired that an image processing device can appropriately set an image area in a frame image.

[0008] It is desirable to provide an image processing device capable of appropriately setting an image region.

[0009] Technical Solution

[0010] An image processing device according to one embodiment of the present invention includes a representative distance calculation unit and a combination processing unit. The representative distance calculation unit is configured to generate a plurality of representative distance values based on a distance image, wherein the distance image is generated based on a stereo image and includes distance values for each pixel. The plurality of representative distance values correspond to a plurality of pixel columns in the distance image and are representative values of the distance values for the corresponding pixel columns. The combination processing unit is configured to perform a combination process for combining a first image region, which is defined based on an image of one or more objects included in the stereo image and including another vehicle, with a second image region. The combination processing unit performs a determination process for determining whether a virtual combination region, which is a region obtained by virtually combining the first and second image regions, includes an image of a corner portion of another vehicle. If the virtual combination region includes an image of a corner portion, the combination process is avoided. The combined processing unit generates a first approximate straight line in a first image region based on a plurality of first representative distance values corresponding to a plurality of pixel columns belonging to the first image region among a plurality of representative distance values, and calculates a first degree of deviation of the plurality of first representative distance values relative to the first approximate straight line. In the second image region, the combined processing unit generates a second approximate straight line based on a plurality of second representative distance values corresponding to a plurality of pixel columns belonging to the second image region among a plurality of representative distance values, and calculates a second degree of deviation of the plurality of second representative distance values relative to the second approximate straight line. The combined processing unit performs determination processing based on the first degree of deviation and the second degree of deviation.

[0011] Technical Effects

[0012] According to an image processing device according to one embodiment of the present invention, a first approximate straight line is generated based on a plurality of first representative distance values corresponding to a plurality of pixel columns belonging to a first image area, and a first degree of deviation of the plurality of first representative distance values relative to the first approximate straight line is calculated. A second approximate straight line is generated based on a plurality of second representative distance values corresponding to a plurality of pixel columns belonging to a second image area, and a second degree of deviation of the plurality of second representative distance values relative to the second approximate straight line is calculated. Determination processing is performed based on the first degree of deviation and the second degree of deviation, thereby enabling appropriate setting of the image area. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a block diagram showing a configuration example of an image processing device according to an embodiment of the present invention.

[0014] Figure 2 This is an explanatory diagram showing an example of a distance image.

[0015] Figure 3 It is an explanatory diagram showing an example of representative distance values.

[0016] Figure 4 It shows Figure 1 An image diagram showing an example of the operation of the processing unit shown.

[0017] Figure 5 It shows Figure 1 A flowchart of an operation example of the processing unit shown.

[0018] Figure 6 It shows Figure 1 Another image diagram of an operation example of the processing unit shown.

[0019] Figure 7 It shows Figure 1 Flowchart of an example of the segmentation processing by the segmentation processing unit shown.

[0020] Figure 8 It shows Figure 1 FIG. 1 is an explanatory diagram of an operation example of the segmentation processing unit shown in FIG.

[0021] Figure 9A It shows Figure 1 Another explanatory diagram of an operation example of the segmentation processing unit shown.

[0022] Figure 9B It shows Figure 1 Another explanatory diagram of an operation example of the segmentation processing unit shown.

[0023] Figure 9C It shows Figure 1 Another explanatory diagram of an operation example of the segmentation processing unit shown.

[0024] Figure 9D It shows Figure 1 FIG. 2 is another explanatory diagram of an operation example of the segmentation processing unit shown in FIG.

[0025] Figure 9E It shows Figure 1 Another explanatory diagram of an operation example of the segmentation processing unit shown.

[0026] Figure 10 It shows Figure 1 Another explanatory diagram of an operation example of the segmentation processing unit shown.

[0027] Figure 11 It shows Figure 1 FIG. 1 is a flowchart of an example of a joining process by the joining processing unit shown in FIG.

[0028] Figure 12 It shows Figure 1 FIG. 1 is an explanatory diagram of an operation example of the combination processing unit shown.

[0029] Figure 13 It shows Figure 1Another explanatory diagram of an operation example of the combined processing unit shown.

[0030] Figure 14 It shows Figure 1 Another explanatory diagram of an operation example of the combined processing unit shown.

[0031] Figure 15 It shows Figure 1 Another explanatory diagram of an operation example of the combined processing unit shown.

[0032] Figure 16A It shows Figure 1 Another image diagram of an operation example of the processing unit shown.

[0033] Figure 16B It shows Figure 1 Another image diagram of an operation example of the processing unit shown.

[0034] Figure 17A It shows Figure 1 A flowchart of an example of another joining process of the joining processing unit shown.

[0035] Figure 17B It shows Figure 1 A flowchart of an example of another joining process of the joining processing unit shown.

[0036] Figure 18 It shows Figure 1 Another explanatory diagram of an operation example of the combined processing unit shown.

[0037] Figure 19 It shows Figure 1 Another explanatory diagram of an operation example of the combined processing unit shown.

[0038] Figure 20 It shows Figure 1 Another explanatory diagram of an operation example of the combined processing unit shown.

[0039] Figure 21 It shows Figure 1 Another explanatory diagram of an operation example of the combined processing unit shown.

[0040] Explanation of symbols

[0041] 1…Image processing device, 10…Vehicle, 11…Stereo camera, 11L…Left camera, 11R…Right camera, 20…Processing unit, 21…Distance image generation unit, 22…Representative distance calculation unit, 23…Image area setting unit, 24…Clustering processing unit, 25…Segmentation processing unit, 26…Joining processing unit, A…Area segmentation processing, A1…Segmentation processing, B…Area joining processing, B1, B2…Joining processing, CP…Corner position, DA…Inter-area distance, DB…Inter-cluster distance, H…Histogram, L…Approximate line, M…Line, P…Pixel, PIC…Stereo image, POS…Virtual segmentation position, POSC…Segmentation position, PL…Left image, PR…Right image, PZ…Distance image, R, R1 to R9…Image area, V…Discrete value, Zmeas…Measured distance value, Zpeak…Representative distance value DETAILED DESCRIPTION

[0042] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0043] <Implementation Method>

[0044] [Example of configuration]

[0045] Figure 1 1 is a diagram showing a configuration example of an image processing device (image processing device 1) according to one embodiment. The image processing device 1 includes a stereo camera 11 and a processing unit 20. The image processing device 1 is mounted on a vehicle 10 such as an automobile.

[0046] The stereo camera 11 is configured to capture the front of the vehicle 10, thereby generating a pair of images (a left image PL and a right image PR) with parallax between them. The stereo camera 11 includes a left camera 11L and a right camera 11R. Each of the left camera 11L and the right camera 11R includes a lens and an image sensor. In this example, the left camera 11L and the right camera 11R are located within the vehicle 10, near the upper portion of the windshield, separated by a predetermined distance in the width direction of the vehicle 10. The left camera 11L and the right camera 11R perform imaging operations in synchronization with each other. The left camera 11L generates a left image PL, and the right camera 11R generates a right image PR. The left image PL and the right image PR constitute a stereo image PIC. The left image PL and the right image PR have parallax between them. The stereo camera 11 generates a series of stereo images PIC by capturing images at a predetermined frame rate (e.g., 60 fps).

[0047] The processing unit 20 is configured to recognize objects by setting image regions R on various objects, such as vehicles in front of the vehicle 10, included in the left image PL and the right image PR, based on the stereo image PIC supplied from the stereo camera 11. In the vehicle 10, for example, information about the objects recognized by the processing unit 20 can be used to control the vehicle's driving or display the recognized object information on an operation display screen. The processing unit 20 is comprised of, for example, a CPU (Central Processing Unit) that executes programs, a RAM (Random Access Memory) that temporarily stores processed data, and a ROM (Read Only Memory) that stores programs. The processing unit 20 includes a distance image generation unit 21, a representative distance calculation unit 22, an image region setting unit 23, a clustering unit 24, a segmentation unit 25, and a combination unit 26.

[0048] The range image generating unit 21 is configured to generate a range image PZ by performing predetermined image processing including stereo matching processing and / or filtering processing based on the left image PL and the right image PR.

[0049] Figure 2 This figure shows an example of a distance image PZ. Similar to the left image PL and the right image PR, the horizontal axis of the distance image PZ corresponds to the x-coordinate in the vehicle width direction of the vehicle 10, and the vertical axis corresponds to the y-coordinate in the vehicle height direction of the vehicle 10. The pixel value of each pixel P in the distance image PZ corresponds to the distance value (measured distance value Zmeas) to the point corresponding to each pixel in three-dimensional real space. In other words, the pixel value of each pixel P corresponds to the z-coordinate in the vehicle length direction of the vehicle 10.

[0050] The representative distance calculation unit 22 is configured to calculate a representative distance value Zpeak for each pixel row in the distance image PZ based on the distance image PZ. Specifically, the representative distance calculation unit 22 sequentially selects multiple pixel rows in the distance image PZ and generates a histogram H based on the multiple measured distance values Zmeas belonging to the selected pixel rows. The representative distance calculation unit 22 then calculates the representative distance value Zpeak for the selected pixel rows based on the histogram H.

[0051] Figure 3 is shown based on Figure 2The diagram shows an example of a histogram Hj generated from multiple measured distance values Zmeas for the j-th pixel row in the distance image PZ. The horizontal axis represents the value of the coordinate z, and the vertical axis represents the frequency. The representative distance calculation unit 22 obtains the coordinate z value with the highest frequency as the representative distance value Zpeak. By generating such a histogram H for all pixel rows, the representative distance calculation unit 22 calculates the representative distance value Zpeak.

[0052] The image region setting unit 23 is configured to set a plurality of image regions R based on the distance image PZ. Specifically, the image region setting unit 23 sets the image region R so that a plurality of pixels P having consecutive measured distance values Zmeas in the distance image PZ belong to one image region R.

[0053] Figure 4 is a diagram illustrating an example of an image region R. In this example, the left image PL in the stereoscopic image PIC includes images of two vehicles 9A and 9B in front of the host vehicle 10. It should be noted that while the left image PL is used in this example, the same applies to the right image PR and the distance image PZ.

[0054] Vehicle 9A is traveling along the road ahead of vehicle 10, and left image PL includes an image of the rear side of vehicle 9A. Because the measured distance values Zmeas in distance image PZ are continuous at the rear side of vehicle 9A, image region setting unit 23 sets image region R (image region RA) that includes the rear side of vehicle 9A.

[0055] Vehicle 9B enters the front of vehicle 10 from the side. Since vehicle 9B is facing in a direction deviating from the direction in which the road extends, left image PL includes an image of the side 101 of vehicle 9B, an image of the back 102 of vehicle 9B, and an image of the corner 103 between the side 101 and back 102 of vehicle 9B. In this example, since the measured distance values Zmeas in distance image PZ are continuous across the side 101, back 102, and corner 103 of vehicle 9B, the image region setting unit 23 sets a single image region R (image region R1) encompassing the side 101, back 102, and corner 103 of vehicle 9B.

[0056] The clustering processing unit 24 is configured to set a single image region R by combining a plurality of image regions R, referring to processing based on a previous stereo image PIC, during processing based on a current stereo image PIC. Specifically, for example, if a plurality of image regions R are set for a certain vehicle in the current stereo image PIC, and a single image region R was set for the same vehicle in processing based on a previous stereo image PIC, the clustering processing unit 24 sets the single image region R by combining these plurality of image regions R.

[0057] The segmentation processing unit 25 is configured to perform a region segmentation process A for segmenting the image region R into two image regions R based on a plurality of representative distance values Zpeak in the image region R when a predetermined condition is satisfied. The region segmentation process A includes a segmentation process A1. In the segmentation process A1, the segmentation processing unit 25 determines whether the image region R1 includes an image of a corner portion of a vehicle based on a plurality of representative distance values Zpeak corresponding to a plurality of pixel columns belonging to a certain image region R (image region R1), for example. Furthermore, if the image region R1 includes an image of the corner portion, the segmentation processing unit 25 segments the image region R1 into two image regions R (image regions R2 and R3) based on the corner portion. That is, in the case where the image region R1 includes an image of the corner portion. Figure 4 In the example, the segmentation processing unit 25 determines that the image area R1 includes the corner portion 103 of the vehicle 9B, thereby segmenting the image area R1 into an image area R2 including an image of the side 101 of the vehicle 9B and an image area R3 including an image of the back 102 of the vehicle 9B.

[0058] The combining processing unit 26 is configured to perform a region combining process B for combining the two image regions R based on a plurality of representative distance values Zpeak of each of the two image regions R when a predetermined condition is satisfied.

[0059] The region combining process B includes combining process B1. In combining process B1, when a portion of the three-dimensional real space corresponding to an image included in a certain image region R (image region R4) and a portion of the three-dimensional real space corresponding to an image included in another image region R (image region R5) are close to each other, combining these two image regions R4 and R5 to form a single image region R (image region R6). Furthermore, combining process 26 determines, for example, whether the region formed by the virtual combination of the two image regions R4 and R5 includes an image of a corner of a vehicle. Furthermore, if the region includes an image of the corner, combining process 26 does not combine the two image regions R4 and R5.

[0060] Furthermore, the region combining process B includes combining process B2. In combining process B2, even if the portion of the three-dimensional real-space image included in a certain image region R (image region R7) and the portion of the three-dimensional real-space image included in another image region R (image region R8) are slightly separated from each other, combining these two image regions R7 and R8 to form a single image region R (image region R9) if predetermined conditions are met. For example, combining process 26 determines whether two separate image regions R7 and R8 both include images of different portions of the side of a particular vehicle. Furthermore, if both image regions R7 and R8 include images of different portions of the side of the vehicle, combining process 26 sets image region R9 by combining these two image regions R7 and R8. Specifically, as will be described later, for example, image region R7 may include an image of the front portion of the side of a long vehicle, such as a bus, while image region R8 may include an image of the rear portion of the side of the vehicle. For example, because pattern matching is difficult on the side of a bus, it is difficult for the range image generation unit 21 to generate a highly accurate range image PZ. In this case, independent image regions R7 and R8 can be set for the front and rear sides of the bus, respectively. In this case, the combination processing unit 26 can combine these two image regions R7 and R8 to create a single image region R9.

[0061] Thus, the processing unit 20 recognizes the vehicle ahead of the vehicle 10. The processing unit 20 then outputs information on the recognition result.

[0062] With this configuration, in image processing device 1, segmentation processing unit 25, in segmentation processing A1, divides image region R (image region R1) into image region R (image region R2) including the side image of the vehicle and image region R (image region R3) including the back image of the vehicle when a certain image region R includes the image of the corner of the vehicle. In combination processing B1, combination processing unit 26 does not combine image region R2 and image region R3 because the region formed by virtually combining these image regions R2 and R3 includes the image of the corner of the vehicle. Thus, in image processing device 1, independent image regions R2 and R3 can be set for the side and back of the vehicle, respectively, allowing recognition of the vehicle separately from the side and back.

[0063] Furthermore, in the image processing device 1, the combining unit 26 combines image regions R (image region R7) and R8 to create a single image region R (image region R9) in combining process B2, when a certain image region R (image region R7) includes an image of the front side of a bus, and another image region R (image region R8) includes an image of the rear side of the bus. Thus, the image processing device 1 can create a single image region R9 for the side of a vehicle, and thus can recognize the side of the vehicle as a single unified image.

[0064] Here, the representative distance calculation unit 22 corresponds to a specific example of the "representative distance calculation unit" in the present invention. The measured distance value Zmeas corresponds to a specific example of the "distance value" in the present invention. The representative distance value Zpeak corresponds to a specific example of the "representative distance value" in the present invention. The image area setting unit 23 corresponds to a specific example of the "image area setting unit" in the present invention. The combination processing unit 26 corresponds to a specific example of the "combination processing unit" in the present invention. The combination processing B1 corresponds to a specific example of the "combination processing" in the present invention. The image area R4 and the image area R5 correspond to a specific example of the "first image area" and the "second image area" in the present invention. The segmentation processing unit 25 corresponds to a specific example of the "segmentation processing unit" in the present invention. The segmentation processing A1 corresponds to a specific example of the "segmentation processing" in the present invention. The image area R1 corresponds to a specific example of the "third image area" in the present invention.

[0065] [Action and Effect]

[0066] Next, the operation and effects of the image processing device 1 according to this embodiment will be described.

[0067] (Overall action summary)

[0068] First, refer to Figure 1 The overall operation of the image processing device 1 will now be described. The stereo camera 11 captures the front of the vehicle 10 and generates a stereo image PIC comprising a left image PL and a right image PR. Based on the stereo image PIC supplied from the stereo camera 11, the processing unit 20 sets image regions R for each of the various objects, such as vehicles in front of the vehicle 10, included in the left and right images PL, thereby recognizing the objects.

[0069] (Detailed actions)

[0070] Figure 5This is a flowchart illustrating an example of the operation of the processing unit 20. Each time a stereo image PIC is supplied from the stereo camera 11, the processing unit 20 generates a distance image PZ based on the stereo image PIC and sets multiple image regions R based on the distance image PZ. Furthermore, the processing unit 20 sets one or more image regions R by dividing one image region R or combining two image regions R. This processing is described in detail below.

[0071] First, the distance image generation unit 21 generates a distance image PZ by performing predetermined image processing including stereo matching processing and / or filtering processing based on the left image PL and the right image PR included in the stereo image PIC (step S101 ).

[0072] Next, the representative distance calculation unit 22 calculates the representative distance value Zpeak for each pixel column in the distance image PZ based on the distance image PZ (step S102). Specifically, the representative distance calculation unit 22 sequentially selects a plurality of pixel columns in the distance image PZ and generates a histogram H ( Figure 3 ). Then, the representative distance calculation unit 22 obtains the value of the coordinate z with the highest frequency in the histogram H as the representative distance value Zpeak. The representative distance calculation unit 22 generates the histogram H for all pixel columns, and calculates the representative distance value Zpeak for each pixel column.

[0073] Next, the image region setting unit 23 sets a plurality of image regions R based on the distance image PZ (step S103 ). Specifically, the image region setting unit 23 sets the image region R so that a plurality of pixels P having consecutive measured distance values Zmeas in the distance image PZ belong to one image region R.

[0074] Next, the clustering processing unit 24, in processing based on the current stereo image PIC, refers to processing based on the previous stereo image PIC and sets a single image region R by combining multiple image regions R (step S104). Specifically, for example, if multiple image regions R are set for a particular vehicle in the current stereo image PIC, and a single image region R was set for the vehicle in processing based on the previous stereo image PIC, the clustering processing unit 24 sets a single image region R by combining these multiple image regions R.

[0075] Next, the segmentation processing unit 25 performs region segmentation processing A (step S105 ) to segment the image region R into two image regions R based on the plurality of representative distance values Zpeak of the image region R if predetermined conditions are satisfied. The segmentation processing A1 included in the region segmentation processing A will be described in detail later.

[0076] Next, the combining unit 26 performs a region combining process B (step S106) to combine the two image regions R based on the plurality of representative distance values Zpeak of each of the two image regions R, if predetermined conditions are satisfied. The combining processes B1 and B2 included in the region combining process B will be described in detail later.

[0077] The above process ends.

[0078] (Division Processing A1)

[0079] Next, Figure 5 The segmentation process A1 included in the area segmentation process A shown in step S105 will be described in detail.

[0080] In segmentation processing A1, the segmentation processing unit 25 determines whether a certain image region R (image region R1) includes an image of a corner of the vehicle based on a plurality of representative distance values Zpeak corresponding to a plurality of pixel columns belonging to the image region R1. If the image region R1 includes an image of the corner, the segmentation processing unit 25 segments the image region R1 into two image regions R (image regions R2 and R3) based on the corner.

[0081] Figure 6 This shows that the segmentation process A1 is used to segment the Figure 4 An example of setting the image areas R2 and R3 based on the image area R1 shown in FIG. Figure 4 As shown, the image region R1 includes an image of the side 101 of the vehicle 9B, an image of the back 102 of the vehicle 9B, and an image of a corner 103 of the vehicle 9B. The segmentation processing unit 25 determines that the image region R1 includes the image of the corner. Then, the segmentation processing unit 25 determines that the image region R1 includes the image of the corner. Figure 6 As shown, image region R1 is divided into image region R2 including an image of the side surface 101 of vehicle 9B and image region R3 including an image of the rear surface 102 of vehicle 9B. Thus, processing unit 20 can recognize vehicle 9B separately from the side surface and the rear surface.

[0082] Figure 7 1 is a diagram showing an example of the division process A1.

[0083] First, the segmentation processing unit 25 selects an image region R (image region R1 ) to be processed among a plurality of image regions R (step S111 ).

[0084] Next, the segmentation processing unit 25 calculates an approximate straight line L using the least squares method based on the plurality of representative distance values Zpeak in the image region R1 , and calculates discrete values of the plurality of representative distance values Zpeak with respect to the approximate straight line L (step S112 ).

[0085] Figure 8 This is a diagram showing an example of the processing in step S112. The horizontal axis represents the coordinate x, and the vertical axis represents the coordinate z. Figure 8 In this xz coordinate plane, multiple coordinate points representing multiple representative distance values Zpeak corresponding to multiple pixel columns belonging to image region R1 are plotted. Based on these multiple representative distance values Zpeak, segmentation processing unit 25 calculates an approximate line L0 using the least squares method. Segmentation processing unit 25 then calculates discrete values V0 for these multiple representative distance values Zpeak relative to approximate line L.

[0086] Next, the segmentation processing unit 25 sets a virtual segmentation position POS in the image region R1 and virtually segments the image region R1 (step S113). In each virtually segmented region, the least squares method is used to calculate an approximate straight line L based on the plurality of representative distance values Zpeak of the region, and the discrete values V of the plurality of representative distance values Zpeak relative to the approximate straight line L are calculated (step S114). If the virtual segmentation has not been performed the predetermined number N times ("No" in step S115), the process returns to step S113. Steps S113 to S115 are then repeated until the predetermined number N of virtual segmentations has been performed.

[0087] Figures 9A to 9E 113 to 115. In this example, the predetermined number N is "5", and the segmentation processing unit 25 sets five virtual segmentation positions POS (virtual segmentation positions POS1 to POS5) in the image region R1.

[0088] For example, Figure 9A As shown, the segmentation processing unit 25 performs virtual segmentation at the virtual segmentation position POS1. Then, based on the multiple representative distance values Zpeak in the area to the left of the virtual segmentation position POS1, the segmentation processing unit 25 calculates an approximate straight line L11 using the least squares method, and calculates discrete values V11 with respect to the approximate straight line L11 for these multiple representative distance values Zpeak. Furthermore, based on the multiple representative distance values Zpeak in the area to the right of the virtual segmentation position POS1, the segmentation processing unit 25 calculates an approximate straight line L12 using the least squares method, and calculates discrete values V12 with respect to the approximate straight line L12 for these multiple representative distance values Zpeak.

[0089] Similarly, the division processing unit 25 divides the virtual division position POS2 ( Figure 9B ) is virtually split and approximate straight lines L21, L22 and discrete values V21, V22 are calculated. By performing a virtual split at the virtual split position POS3 ( Figure 9C ) is virtually split and approximate straight lines L31, L32 and discrete values V31, V32 are calculated. By performing a virtual split at the virtual split position POS4 ( Figure 9D ) is virtually split and approximate straight lines L41, L42 and discrete values V41, V42 are calculated. By performing a virtual split at the virtual split position POS5 ( Figure 9E ) and perform virtual division to calculate approximate straight lines L51, L52 and discrete values V51, V52.

[0090] Next, the segmentation processing unit 25 selects the virtual segmentation position POS that minimizes the degree of deviation from the approximate straight line L (step S116 ). The degree of deviation from the approximate straight line L can be evaluated using the following formula, for example.

[0091]

Formula 1

[0092]

[0093] Here, σ1 represents the degree of deviation from approximate lines L11 and L12 when a virtual split is performed at virtual split position POS1. This degree of deviation σ1 is the sum of the square root of the discrete value V11 in the area to the left of virtual split position POS1 and the square root of the discrete value V12 in the area to the right of virtual split position POS1. In other words, this degree of deviation σ1 is the sum of the standard deviation in the area to the left of virtual split position POS1 and the standard deviation in the area to the right of virtual split position POS1. Similarly, σ2 represents the degree of deviation from approximate lines L21 and L22 when a virtual split is performed at virtual split position POS2. σ3 represents the degree of deviation from approximate lines L31 and L32 when a virtual split is performed at virtual split position POS3. σ4 represents the degree of deviation from approximate lines L41 and L42 when a virtual split is performed at virtual split position POS4. σ5 represents the degree of deviation from approximate lines L51 and L52 when a virtual split is performed at virtual split position POS5. The segmentation processing unit 25 selects the virtual segmentation position POS that obtains the smallest deviation degree σ among the deviation degrees σ1 to σ5. Figures 9A to 9E In the example of , the segmentation processing unit 25 selects the virtual segmentation position POS4.

[0094] Next, the division processing unit 25 estimates the position of the corner portion of the vehicle (corner position CP) based on the virtual division position POS selected in step S116 (step S117 ).

[0095] Figure 10 117 is a diagram showing an example of the processing of step S117. In this example, the segmentation processing unit 25 extends the approximate straight line L41 in the left area of the selected virtual segmentation position POS4 to the right area of the virtual segmentation position POS4, and at the same time, extends the approximate straight line L42 in the right area of the selected virtual segmentation position POS4 to the left area of the virtual segmentation position POS4. The position where the extended approximate straight line L41 and the extended approximate straight line L42 intersect each other corresponds to the corner position CP. That is, since the horizontal axis represents the coordinate x in the vehicle width direction of the vehicle 10 as the host vehicle, and the vertical axis represents the coordinate z in the vehicle length direction of the vehicle 10, on the xz plane, the approximate straight line L41 corresponds to Figure 4 In the illustrated side surface 101 of vehicle 9B, approximate line L42 corresponds to rear surface 102 of vehicle 9B. Thus, the intersection of extended approximate lines L41 and L42 corresponds to the position of corner 103 of vehicle 9B. In this manner, segmentation processing unit 25 estimates corner position CP.

[0096] Next, the segmentation processing unit 25 determines whether predetermined segmentation conditions for segmenting the image region R1 are satisfied based on the corner position CP estimated in step S117 (step S118). The predetermined segmentation conditions may include, for example, the following five conditions. The segmentation processing unit 25 determines that the predetermined segmentation conditions are satisfied when these five conditions are satisfied.

[0097] The first condition is that the x coordinate of the corner position CP estimated in step S117 is located in the image region R1. That is, since satisfying this first condition indicates that the image region R1 is likely to include an image of a corner portion, this first condition can serve as a segmentation condition.

[0098] The second condition is that, when image region R1 is segmented at the x-coordinate of corner position CP, the discrete value V relative to the approximate straight line L is smaller than the discrete value V0 calculated in step S112. In other words, since satisfying this second condition indicates that the two image regions R (image regions R2 and R3) are more appropriately set by segmentation, this second condition can serve as a segmentation condition.

[0099] The third condition is the angle θ ( Figure 10) is within a predetermined angle range including 90 degrees. That is, because the two approximate straight lines L correspond to the side and back of the vehicle, it is expected that the angle θ formed by the two approximate straight lines L will be approximately 90 degrees. Since satisfying this third condition indicates a high probability that image region R1 includes an image of a corner portion, this third condition can serve as a segmentation condition.

[0100] The fourth condition is that the width of the area to the left of the x-coordinate of the corner position CP in the image region R1 is greater than or equal to a predetermined width, and the width of the area to the right of the x-coordinate of the corner position CP is greater than or equal to a predetermined width. In other words, since satisfying this fourth condition indicates that the sizes of the two image regions R (image regions R2 and R3) set by division are of a certain size, this fourth condition can serve as a division condition.

[0101] The fifth condition is that the discrete value V0 calculated in step S112 is equal to or greater than a predetermined value. In other words, satisfying this fifth condition indicates that approximation using a single approximate straight line L0 is inappropriate. In other words, there is a high probability that image region R1 includes an image at a corner. Therefore, this fifth condition can serve as a segmentation condition.

[0102] When all of these multiple conditions are satisfied, the segmentation processing unit 25 determines that the predetermined segmentation condition is satisfied. In other words, the segmentation processing unit 25 determines that the image region R1 includes an image of a corner portion.

[0103] If the predetermined segmentation condition is not satisfied in step S118 ("No" in step S118), the process ends. If the predetermined segmentation condition is satisfied ("Yes" in step S118), the segmentation processing unit 25 sets the x-coordinate of the corner position CP estimated in step S117 as the segmentation position POSC and uses this segmentation position POSC to segment the image region R1 into the two image regions R2 and R3 (step S119).

[0104] The segmentation processing A1 is completed as described above. The segmentation processing unit 25 sequentially selects an image region R1 to be processed from among the plurality of image regions R, and performs the segmentation processing A1 on the selected image region R1.

[0105] In the image processing apparatus 1, as Figure 4As shown, since image region R1 includes an image of a vehicle corner, the predetermined segmentation condition is satisfied ("Yes" in step S118), and therefore the segmentation processing unit 25 segments image region R1 into image region R2 including an image of the vehicle's side surface and image region R3 including an image of the vehicle's rear surface. Thus, in the image processing device 1, independent image regions R2 and R3 can be set for the vehicle's side surface and rear surface, respectively, enabling recognition of the vehicle by separating the vehicle's side surface and rear surface.

[0106] (Combined with treatment B1)

[0107] Next, Figure 5 The combining process B1 included in the region combining process B shown in step S106 will be described in detail.

[0108] In this combination process B1, when the portion of the three-dimensional real space corresponding to the image included in a certain image region R (image region R4) and the portion of the three-dimensional real space corresponding to the image included in another image region R (image region R5) are close to each other, the combination processing unit 26 combines these two image regions R4 and R5 to form a single image region R (image region R6). Furthermore, the combination processing unit 26 determines, for example, whether the region resulting from the virtual combination of the two image regions R4 and R5 includes an image of a vehicle corner. If this region includes an image of the corner, the combination processing unit 26 avoids combining the two image regions R4 and R5.

[0109] Figure 11 It is a diagram showing an example of the combining process B1.

[0110] First, the combination processing unit 26 selects two image regions R (image regions R4 and R5) to be processed from among the plurality of image regions R (step S121). Specifically, the combination processing unit 26 selects two image regions R4 and R5 from among the plurality of image regions R whose distance in the three-dimensional real space corresponding to the image included in the image region R is closer than a predetermined distance.

[0111] Next, the combination processing unit 26 virtually combines the image regions R4 and R5 and calculates an approximate straight line L using the least squares method based on a plurality of representative distance values Zpeak in the virtually combined regions (step S122 ).

[0112] Figure 12 1 is a diagram showing an example of the processing in step S122. Figure 12In the xz coordinate plane, coordinate points representing a plurality of representative distance values Zpeak corresponding to a plurality of pixel columns belonging to the image region R1 are plotted. The concatenation processing unit 26 calculates an approximate straight line L100 using the least squares method based on these plurality of representative distance values Zpeak.

[0113] Next, the combination processing unit 26 calculates an approximate straight line L based on the plurality of representative distance values Zpeak using the least squares method in each of the image regions R4 and R5 , and calculates discrete values V of the plurality of representative distance values Zpeak with respect to the approximate straight line L (step S123 ).

[0114] Figure 13 This figure illustrates an example of the processing in step S123. The combined processing unit 26 calculates an approximate line L101 using the least squares method based on the plurality of representative distance values Zpeak in image region R4, and calculates discrete values V101 relative to the approximate line L101 for these plurality of representative distance values Zpeak. Similarly, the combined processing unit 26 calculates an approximate line L102 using the least squares method based on the plurality of representative distance values Zpeak in image region R5, and calculates discrete values V102 relative to the approximate line L102 for these plurality of representative distance values Zpeak.

[0115] Next, the combination processing unit 26 calculates a straight line M in each of the image regions R4 and R5 using the inclination and intercept of the approximate straight line L calculated in step S122, and calculates a plurality of discrete values V representing the distance values Zpeak relative to the straight line M (step S124).

[0116] Figure 14 is a diagram illustrating an example of the processing in step S124. In image region R4, the combined processing unit 26 calculates a straight line M111 using the inclination and intercept of the approximate straight line L calculated in step S122, and thereby calculates a plurality of discrete values V111 for the representative distance values Zpeak relative to this straight line M111. These discrete values V111 are the discrete values V for the representative distance values Zpeak relative to the approximate straight line L100 in image region R4. Similarly, in image region R5, the combined processing unit 26 calculates a straight line M112 using the inclination and intercept of the approximate straight line L calculated in step S122, and calculates a plurality of discrete values V112 for the representative distance values Zpeak relative to this straight line M112. These discrete values V112 are the discrete values V for the representative distance values Zpeak relative to the approximate straight line L100 in image region R5.

[0117] Next, the combining processing unit 26 calculates a plurality of discrete values V of representative distance values Zpeak with respect to the approximate straight line L calculated in step S122 in the virtually combined region (step S125 ).

[0118] Figure 15 : is a diagram showing an example of the process of step S125. The combining processing unit 26 calculates a plurality of discrete values V100 representing the distance value Zpeak with respect to the approximate straight line L100 calculated in step S122 in the virtually combined region.

[0119] Next, the combination processing unit 26 calculates the change rate of the discrete value V obtained by the virtual combination (step S126). The change rate of the discrete value V can include the following three change rates, for example.

[0120] The first change rate is a change rate of the discrete value V in the entire region obtained by virtual combination. Specifically, the first change rate is, for example, the change rate of the discrete value V101 in the image region R4 and the discrete value V102 in the image region R5 ( Figure 13 ) to the discrete value V100 ( Figure 15 ) rate of change.

[0121] The second change rate is a change rate of the discrete value V in the image region R4 obtained by virtual combination. Specifically, the second change rate is, for example, a change rate from the discrete value V101 ( Figure 13 ) to the discrete value V111 ( Figure 14 ) rate of change.

[0122] The third change rate is a change rate of the discrete value V in the image region R5 obtained by virtual combination. Specifically, the third change rate is, for example, a change rate from the discrete value V102 ( Figure 13 ) to the discrete value V112 ( Figure 14 ) rate of change.

[0123] Next, the combination processing unit 26 determines whether the predetermined combination conditions for combining image regions R4 and R5 are met (step S127). The predetermined combination conditions can include, for example, the following three conditions. The combination processing unit 26 determines that the predetermined combination conditions are met when these three conditions are met. The first condition is that the first rate of change is less than a predetermined first threshold. The second condition is that the second rate of change is less than a predetermined second threshold. The third condition is that the third rate of change is less than a predetermined third threshold. The combination processing unit 26 determines whether the predetermined combination conditions are met by determining whether all of these multiple conditions are met.

[0124] If the predetermined combining conditions are not satisfied in step S127 ("No" in step S127), the process ends. If the predetermined combining conditions are satisfied ("Yes" in step S127), the combining processing unit 26 combines image region R4 and image region R5 to set image region R6 (step S128).

[0125] The combining process B1 is completed as described above. The combining process unit 26 sequentially selects two image regions R4 and R5 to be processed, and performs the combining process B1 on the selected two image regions R4 and R5.

[0126] In the image processing apparatus 1, for example, Figure 7 As described in segmentation process A1, when image region R1 includes an image of a vehicle corner, image region R1 is segmented into image region R2, which includes an image of the vehicle's side, and image region R3, which includes an image of the vehicle's back. In this case, in join process B1, if image regions R2 and R3 are joined, the discrete value V increases. Since the predetermined join condition is not met ("No" in step S127), join process 26 does not join image regions R2 and R3. The predetermined join condition is that the virtually joined region does not include an image of the vehicle's corner. That is, in the example of image regions R2 and R3, since the predetermined join condition is not met, the virtually joined region is determined to include an image of the vehicle's corner. Consequently, join process 26 does not join image regions R2 and R3. This prevents image regions R2 and R3, which were segmented and set in segmentation process A1, from being joined in join process B1.

[0127] (Combined with Process B2)

[0128] Next, Figure 5 The combining process B2 included in the region combining process B shown in step S106 will be described in detail.

[0129] In this combining process B2, the combining processing unit 26 determines whether two separate image regions R (image regions R7 and R8) both include images of different portions of the side surface of a particular vehicle. If both image regions R7 and R8 include images of different portions of the side surface of the vehicle, the combining processing unit 26 combines the two image regions R7 and R8 to create an image region R9.

[0130] Figure 16A 1 and 2 are diagrams showing examples of image regions R7 and R8. Figure 16B FIG. 1 is a diagram illustrating an example of image region R9 formed by combining image regions R7 and R8 using the combining process B2. In this example, the left image PL in the stereoscopic image PIC includes an image of vehicle 9C in front of vehicle 10, which is the host vehicle. While this example uses the left image PL for explanation, the same applies to the right image PR and the distance image PZ.

[0131] Vehicle 9C is a long vehicle, a bus in this example. Vehicle 9C enters the front of vehicle 10 from the side. Since vehicle 9C is facing a direction deviating from the extension direction of the road, the left image PL includes an image of the side of vehicle 9C. The image area setting unit 23 is expected to set an image area R on the side of vehicle 9C. However, for example, since it is difficult to perform graphic matching on the side of a bus, it is difficult for the distance image generation unit 21 to generate a high-precision distance image PZ. Specifically, it is difficult for the distance image generation unit 21 to calculate the value of the coordinate z, for example, near the center of the side of the bus. Therefore, when the representative distance calculation unit 22 generates a histogram H of a certain pixel column corresponding to the center of the side of the bus in the distance image PZ, the amount of data may be insufficient. In this case, the representative distance calculation unit 22 cannot calculate the representative distance value Zpeak. Therefore, the representative distance value Zpeak is missing in the pixel column corresponding to the center of the side of the bus. As a result, as Figure 16A As shown, the image region setting unit 23 can set two image regions R7 and R8 for the side surface of one vehicle 9C.

[0132] In the combining process B2, the combining processing unit 26 determines that the two image regions R7 and R8 both include images of different portions of the side surface of a certain vehicle. Figure 16B As shown, the combining processing unit 26 combines these image regions R7 and R8 to set the image region R9. This allows the processing unit 20 to recognize the side of the vehicle 9C as a unified single image.

[0133] Figure 17A 、 17Bis a diagram showing an example of the combined process B2. Steps S132 to S137 in the combined process B2 are Figure 11 Steps S122 to S127 in the shown joining process B1 are the same.

[0134] First, the combination processing unit 26 selects two image regions R (image regions R7 and R8) to be processed from the plurality of image regions R (step S131). The combination processing unit 26 selects the two image regions R in order from the left end toward the center of the image, and selects the two image regions R in order from the right end toward the center of the image. Thereafter, the combination processing unit 26 again selects the two image regions R in order from the left end toward the center of the image, and selects the two image regions R in order from the right end toward the center of the image.

[0135] Next, the combination processing unit 26 virtually combines the image regions R7 and R8 and calculates an approximate straight line L using the least squares method based on a plurality of representative distance values Zpeak in the virtually combined regions (step S132 ).

[0136] Figure 18 132 is a diagram showing an example of the processing of step S132. Figure 18 In the xz coordinate plane, coordinate points representing a plurality of representative distance values Zpeak corresponding to a plurality of pixel columns belonging to the image regions R7 and R8 are plotted. Based on these plurality of coordinate points, the combination processing unit 26 calculates an approximate straight line L200 using the least squares method.

[0137] Next, the combination processing unit 26 calculates an approximate straight line L based on the plurality of representative distance values Zpeak using the least squares method in each of the image regions R7 and R8 , and calculates discrete values V of the plurality of representative distance values Zpeak with respect to the approximate straight line L (step S133 ).

[0138] Next, the combination processing unit 26 calculates a straight line M in the image areas R7 and R8 using the inclination and intercept of the approximate straight line L calculated in step S132, and calculates a plurality of discrete values V representing the distance values Zpeak relative to the straight line M (step S134).

[0139] Next, the combining processing unit 26 calculates a plurality of discrete values V of representative distance values Zpeak with respect to the approximate straight line L calculated in step S132 in the virtually combined region (step S135 ).

[0140] Next, the combination processing unit 26 calculates the rate of change of the discrete value V obtained by the virtual combination (step S136 ).

[0141] Next, the combining unit 26 determines whether predetermined combining conditions for combining image regions R7 and R8 are satisfied (step S137 ). If the predetermined combining conditions are not satisfied (“No” in step S137 ), the flow ends.

[0142] In step S137, if the predetermined combination condition is satisfied ("Yes" in step S137), the combination processing unit 26 calculates the inter-region distance DA between the coordinate point representing the representative distance value Zpeak in the image region R7 and the coordinate point representing the representative distance value Zpeak in the image region R8 (step S138). Specifically, Figure 18 As shown, the combined processing unit 26 calculates the distance along the approximate straight line L200 between the coordinate point representing the representative distance value Zpeak in the image area R7 that is closest to the image area R8 and the coordinate point representing the representative distance value Zpeak in the image area R8 that is closest to the image area R7 in the xz coordinate plane as the inter-area distance DA.

[0143] Next, the combining unit 26 determines whether the inter-area distance DA is within a predetermined threshold value Dth (step S139). If the inter-area distance DA is within the predetermined threshold value Dth ("Yes" in step S139), the process proceeds to step S146.

[0144] In step S139, when the inter-area distance DA is not within the predetermined threshold value Dth ("No" in step S139), the processing unit 26 draws the coordinate point Q of the measured distance value Zmeas of each pixel P in the distance image PZ and the measured distance value Zmeas of each pixel P between the image area R7 and the image area R8 in the xz coordinate plane (step S140).

[0145] Figure 19 1 is a diagram showing an example of the processing in step S140. In this example, five coordinate points Q are plotted based on the measured distance values Zmeas of each pixel P between the image region R7 and the image region R8. It should be noted that in this example, five coordinate points Q are plotted for ease of explanation, but in reality, more coordinate points Q can be plotted. These coordinate points Q represent the measured distance values Zmeas generated by the distance image generating unit 21. Figure 19 As shown, it may be slightly away from the approximate straight line L200.

[0146] Next, the combination processing unit 26 moves the coordinate point Q drawn in step S140 onto the approximate straight line L in the xz coordinate plane (step S141 ).

[0147] Figure 20This figure illustrates an example of the processing of step S141. In this example, the combination processing unit 26 moves the coordinate point Q drawn in step S140 toward the approximate line L200 in a direction perpendicular to the approximate line L200. As a result, the moved coordinate point Q is located on the approximate line L200.

[0148] Next, the joining processing unit 26 calculates the density of the plurality of coordinate points Q on the approximate straight line L, and deletes the coordinate points Q in a region with a low density among the plurality of coordinate points Q (step S142 ).

[0149] Figure 21 142 is a diagram showing an example of the process of step S142. In this example, the center coordinate point Q of the five coordinate points Q moved onto the approximate straight line L200 in step S141 is located in a low-density area, so the joining processing unit 26 deletes this coordinate point Q.

[0150] Next, the combination processing unit 26 divides the plurality of coordinate points Q on the approximate straight line L among the plurality of coordinate points Q into the group G7 corresponding to the image region R7 or the group G8 corresponding to the image region R8 (step S143 ).

[0151] Next, the combination processing unit 26 calculates the inter-group distance DB between the coordinate point Q in the group G7 and the coordinate point Q in the group G8 along the approximate straight line L (step S144). Figure 21 As shown, the combination processing unit 26 calculates the distance along the approximate straight line L200 between the coordinate point Q in group G7 that is closest to group G8 and the coordinate point Q in group G8 that is closest to group G7 in the xz plane as the inter-group distance DB.

[0152] Next, the joining processing unit 26 checks whether the inter-cluster distance DB is within a predetermined threshold value Dth (step S145 ). If the inter-cluster distance DB is not within the predetermined threshold value Dth (“No” in step S145 ), the flow ends.

[0153] Then, the combining processing unit 26 combines the image region R7 and the image region R8 to set the image region R9 (step S146 ).

[0154] The combining process B2 is completed as described above. The combining process unit 26 sequentially selects two image regions R7 and R8 to be processed, and performs the combining process B2 on the selected two image regions R7 and R8.

[0155] like Figure 16AAs shown, in the image processing device 1, when two separate image regions R7 and R8 both include images of different portions of the side of a vehicle, the discrete value V can be suppressed to a relatively small value even when these image regions R7 and R8 are combined, thus satisfying the predetermined combination condition ("Yes" in step S137). In this case, for example, if the side portion of the vehicle shown in the image region R7 and the side portion of the vehicle shown in the image region R8 are close to each other, the inter-region distance DA is within the predetermined threshold value Dth ("Yes" in step S139), so the combination processing unit 26 combines the two image regions R7 and R8 to set the image region R9. On the other hand, if the side portion of the vehicle shown in the image region R7 and the side portion of the vehicle shown in the image region R8 are slightly separated, the inter-region distance DA may be longer than the predetermined threshold value Dth ("No" in step S139). Even in this case, the inter-group distance DB is calculated using the measured distance value Zmeas of each pixel P between image regions R7 and R8. If the inter-group distance DB is within a predetermined threshold value Dth ("Yes" in step S145), image region R9 is set by combining the two image regions R7 and R8. Thus, in the image processing device 1, since a single image region R9 can be set for the side of a vehicle, the side of the vehicle can be recognized as a unified image.

[0156] Here, approximate straight line L101 corresponds to a specific example of the "first approximate straight line" in the present invention. Approximate straight line L102 corresponds to a specific example of the "second approximate straight line" in the present invention. Approximate straight line L100 corresponds to a specific example of the "third approximate straight line" in the present invention. Approximate straight line L41 corresponds to a specific example of the "fourth approximate straight line" in the present invention. Approximate straight line L42 corresponds to a specific example of the "fifth approximate straight line" in the present invention. Approximate straight line L0 corresponds to a specific example of the "sixth approximate straight line" in the present invention. Discrete value V101 corresponds to a specific example of the "first degree of deviation" in the present invention. Discrete value V102 corresponds to a specific example of the "second degree of deviation" in the present invention. Discrete value V100 corresponds to a specific example of the "third degree of deviation" in the present invention. Discrete value V111 corresponds to a specific example of the "fourth degree of deviation" in the present invention. Discrete value V112 corresponds to a specific example of the "fifth degree of deviation" in the present invention. Discrete value V0 corresponds to a specific example of the "sixth degree of deviation" in the present invention. Discrete value V41 corresponds to a specific example of the "seventh degree of deviation" in the present invention. Discrete value V42 corresponds to a specific example of the "eighth degree of deviation" in the present invention. Virtual division position POS4 corresponds to a specific example of the "virtual division position" in the present invention. Virtual division positions POS1 to POS5 correspond to a specific example of the "plurality of virtual division positions" in the present invention. Division position POSC corresponds to a specific example of the "division position" in the present invention.

[0157] As described above, in image processing device 1, segmentation processing unit 25 determines whether image region R1 includes an image of a corner portion of another vehicle based on the multiple representative distance values Zpeak corresponding to the multiple pixel columns belonging to image region R1. If image region R1 includes an image of a corner portion, segmentation processing is performed based on the corner portion to set two image regions R2 and R3. Thus, in image processing device 1, independent image regions R2 and R3 can be set for the side and rear of another vehicle, respectively, allowing identification of the other vehicle separately from the side and rear. Thus, image processing device 1 can appropriately set image region R.

[0158] In particular, Figure 10As shown, in the image processing device 1, the segmentation processing unit 25 sets a virtual segmentation position POS (in this example, virtual segmentation position POS4) in the image region R1 and divides the plurality of representative distance values Zpeak into a group to the left of the virtual segmentation position POS and a group to the right of the virtual segmentation position POS. The segmentation processing unit 25 then generates an approximate straight line L (in this example, approximate straight line L41) based on the plurality of representative distance values Zpeak divided into the group to the left of the virtual segmentation position POS, and generates an approximate straight line L (in this example, approximate straight line L42) based on the plurality of representative distance values Zpeak divided into the group to the right of the virtual segmentation position POS, and sets the segmentation position POSC based on these approximate straight lines L. Thus, the image processing device 1 can set the segmentation position POSC with high precision.

[0159] Furthermore, in image processing device 1, the combination processing unit 26 determines whether the area resulting from the virtual combination of image areas R4 and R5 (the virtual combination area) includes an image of the corner of another vehicle. If the virtual combination area includes an image of a corner, the two image areas R4 and R5 are not combined. Thus, in image processing device 1, for example, if image area R4 includes an image of the side of another vehicle and image area R5 also includes an image of the side of another vehicle, these two image areas R4 and R5 are not combined. This allows the recognition of the other vehicle to be separated into its side and rear views. Thus, image processing device 1 can appropriately set image area R.

[0160] In particular, if Figure 13 As shown, in the image processing device 1, the join processing unit 26 generates an approximate line L101 based on the multiple representative distance values Zpeak corresponding to the multiple pixel columns belonging to the image region R4, and calculates discrete values V101 for these multiple representative distance values relative to the approximate line L101. Similarly, the join processing unit 26 generates an approximate line L102 based on the multiple representative distance values Zpeak corresponding to the multiple pixel columns belonging to the image region R5, and calculates discrete values V102 for these multiple representative distance values relative to the approximate line L102. Furthermore, the join processing unit 26 determines whether the virtual join region includes an image of the corner of another vehicle based on these discrete values V. Thus, in the image processing device 1, if the discrete value V increases due to virtual joining, it can be determined that the image includes an image of the corner of another vehicle, and the two image regions R4 and R5 can be prevented from being joined.

[0161] Furthermore, in the image processing device 1, the combining processing unit 26 determines whether both image regions R7 and R8 include an image of the side of another vehicle. If both image regions R7 and R8 include an image of the side of another vehicle, the two image regions R7 and R8 are combined. Thus, in the image processing device 1, for example, if image region R7 includes an image of the front side of another vehicle and image region R8 includes an image of the rear side of another vehicle, the two image regions R7 and R8 can be combined, thereby allowing the side of the other vehicle to be recognized as a unified image. Thus, the image processing device 1 can appropriately set the image region R.

[0162] In particular, Figure 21 As shown, in the image processing device 1, the combination processing unit 26 determines whether both image regions R7 and R8 include an image of the side of another vehicle based on the representative distance value Zpeak corresponding to the plurality of pixel columns belonging to image region R7, the representative distance value Zpeak corresponding to the plurality of pixel columns belonging to image region R8, and the measured distance value Zmeas of each pixel P between image regions R7 and R8. Thus, in the image processing device 1, even if, for example, the pixel column corresponding to the side of a bus near the center lacks the representative distance value Zpeak, the measured distance value Zmeas can be used to easily determine that both image regions R7 and R8 include the same image of the side of another vehicle, and the two image regions R7 and R8 can be easily combined.

[0163] This allows the image processing device 1 to appropriately set the image area. This allows the vehicle 10 to more accurately control the driving of the vehicle 10 based on the recognition result, or to display more accurate information about the recognized object on the operation display screen.

[0164] [Effect]

[0165] As described above, in this embodiment, the image processing device determines whether image region R1 includes an image of a corner portion of another vehicle based on the multiple representative distance values corresponding to the multiple pixel columns belonging to image region R1. If image region R1 includes an image of a corner portion, the image processing device performs segmentation processing based on the corner portion to set two image regions R2 and R3. This allows independent image regions R2 and R3 to be set for the side and rear of the other vehicle, respectively, enabling appropriate image region setting.

[0166] In this embodiment, the image processing device determines whether the virtual combined area, which is the area formed by virtually combining image areas R4 and R5, includes an image of the corner of another vehicle. If the virtual combined area includes an image of the corner, the two image areas R4 and R5 are not combined. Thus, for example, if image area R4 includes the side image of another vehicle and image area R5 includes the rear image of another vehicle, the two image areas R4 and R5 are not combined, thereby enabling appropriate setting of the image areas.

[0167] In this embodiment, the image processing device determines whether both image regions R7 and R8 include an image of the side of another vehicle, and combines the two image regions R7 and R8 if both image regions R7 and R8 do. Thus, for example, if image region R7 includes an image of the front side of another vehicle and image region R8 includes an image of the rear side of the other vehicle, the two image regions R7 and R8 can be combined, thereby enabling appropriate setting of the image regions.

[0168] As mentioned above, although this technology was demonstrated using embodiment as an example, this technology is not limited to these embodiment etc., and various changes are possible.

[0169] For example, in the above embodiment, the stereo camera 11 captures the front of the vehicle 10 , but the present invention is not limited thereto. For example, the stereo camera 11 may capture the side and / or rear of the vehicle 10 .

[0170] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be achieved.

Claims

1. An image processing device, characterized in that have: a representative distance calculation unit for generating a plurality of representative distance values based on a distance image, wherein the distance image is an image generated based on a stereo image and includes a distance value for each pixel, the plurality of representative distance values corresponding to a plurality of pixel columns in the distance image and being representative values of the distance values for the corresponding pixel columns; as well as a combining processing unit capable of combining a first image area and a second image area, which are set based on images of one or more objects included in the stereoscopic image and including the other vehicle, and performing a determination process for determining whether a virtual combined area, which is an area formed by virtually combining the first image area and the second image area, includes an image of a corner portion of the other vehicle, and avoiding the combining process if the virtual combined area includes the image of the corner portion. The combining processing unit generates a first approximate straight line in the first image region based on a plurality of first representative distance values corresponding to a plurality of pixel columns belonging to the first image region among the plurality of representative distance values, and calculates a first degree of deviation of the plurality of first representative distance values from the first approximate straight line. The combining processing unit generates a second approximate straight line in the second image region based on a plurality of second representative distance values corresponding to a plurality of pixel columns belonging to the second image region among the plurality of representative distance values, and calculates a second degree of deviation of the plurality of second representative distance values from the second approximate straight line. The combination processing unit performs the determination processing based on the first deviation degree and the second deviation degree.

2. The image processing device according to claim 1, wherein The image in the first image area shows the side of the other vehicle, The image in the second image area shows the back side of the other vehicle.

3. The image processing device according to claim 1 or 2, characterized in that In the determination process, The combination processing unit generates a third approximate straight line based on the plurality of first representative distance values and the plurality of second representative distance values in the virtual combination area, and calculates a third degree of deviation of the plurality of first representative distance values and the plurality of second representative distance values from the third approximate straight line. The joining processing unit determines whether the virtual joining area includes the image of the corner portion based on the first deviation degree, the second deviation degree, and the third deviation degree.

4. The image processing device according to claim 1 or 2, characterized in that In the determination process, The combining unit generates a third approximate straight line based on the plurality of first representative distance values and the plurality of second representative distance values in the virtual combined area. The combining processing unit calculates a fourth degree of deviation of the plurality of first representative distance values from the third approximate straight line in the first image region. The combining processing unit calculates a fifth degree of deviation of a plurality of the second representative distance values from the third approximate straight line in the second image region. The joining processing unit determines whether the virtual joining area includes the image of the corner portion based on the first deviation degree, the second deviation degree, the fourth deviation degree, and the fifth deviation degree.

5. The image processing device according to claim 1 or 2, characterized in that The image processing device further comprises: an image region setting unit that sets a third image region based on the image of the one or more objects; and A segmentation processing unit is capable of performing segmentation processing to segment the third image area into the first image area and the second image area, performing initial determination processing to determine whether the third image area includes an image of the corner portion of the other vehicle based on a plurality of third representative distance values corresponding to a plurality of pixel columns belonging to the third image area, and performing the segmentation processing based on the corner portion when the third image area includes an image of the corner portion.

6. The image processing device according to claim 5, wherein: In the initial determination process, The segmentation processing unit sets a virtual segmentation position in the third image area, The segmentation processing unit divides the plurality of third representative distance values into a first group and a second group based on the virtual segmentation position, The segmentation processing unit generates a fourth approximate straight line based on a plurality of the third representative distance values classified into the first group among the plurality of the third representative distance values. The segmentation processing unit generates a fifth approximate straight line based on a plurality of the third representative distance values classified into the second group among the plurality of the third representative distance values. The segmentation processing unit sets a segmentation position in the third image region based on the fourth approximate straight line and the fifth approximate straight line.

7. The image processing device according to claim 6, wherein: The segmentation processing unit determines that the third image region includes the image of the corner portion when a predetermined condition is satisfied, The predetermined condition includes satisfying a first condition regarding the segmentation position in the third image region.

8. The image processing device according to claim 7, wherein The segmentation processing unit determines that the third image region includes the image of the corner portion when a predetermined condition is satisfied, The predetermined condition includes satisfying a second condition regarding an angle formed by the fourth approximate straight line and the fifth approximate straight line.

9. The image processing device according to claim 7, wherein: In the initial determination process, The segmentation processing unit generates a sixth approximate straight line based on the plurality of third representative distance values, and calculates a sixth degree of deviation of the plurality of third representative distance values from the sixth approximate straight line. The segmentation processing unit calculates a seventh degree of deviation of the third representative distance values, among the plurality of third representative distance values, classified into the first group, from the fourth approximate straight line. The segmentation processing unit calculates an eighth degree of deviation of the third representative distance values, among the plurality of third representative distance values, classified into the second group, from the fifth approximate straight line. The segmentation processing unit determines that the third image region includes the image of the corner portion when a predetermined condition is satisfied, The predetermined condition includes satisfying a third condition regarding the sixth deviation degree, the seventh deviation degree, and the eighth deviation degree.

10. The image processing device according to claim 6, wherein In the initial determination process, The segmentation processing unit sets a plurality of virtual segmentation positions in the third image area. The segmentation processing unit divides the plurality of third representative distance values into the first group and the second group based on each of the plurality of virtual segmentation positions. The segmentation processing unit generates the fourth approximate straight line for each of the plurality of virtual segmentation positions based on the plurality of third representative distance values classified into the first group among the plurality of third representative distance values, and calculates a seventh degree of deviation of the plurality of third representative distance values classified into the first group from the fourth approximate straight line. The segmentation processing unit generates the fifth approximate straight line for each of the plurality of virtual segmentation positions based on the plurality of third representative distance values classified into the second group among the plurality of third representative distance values, and calculates an eighth degree of deviation of the plurality of third representative distance values classified into the second group from the fifth approximate straight line. The division processing section selects one of the plurality of virtual division positions as the virtual division position based on the seventh deviation degree and the eighth deviation degree in each of the plurality of virtual division positions.

Images