Image processing apparatus, image processing method, mobile device, and computer program

Abstract

To provide an image processing device capable of reducing influence of distance variation.SOLUTION: An image processing device includes: boundary detection means which divides the image information into a plurality of divided regions extending in a predetermined direction and detects the boundary based on at least the color information for an image in each of divided regions; and integrated distance equivalent information determining means for determining the integrated distance equivalent information for each sub-region divided by the boundary in the divided region based on the distance equivalent information of each pixel of the image information.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, a moving body, a computer program, etc. for processing distance information to an object.

Background Art

[0002] There is an imaging device having a sensor in which a plurality of pixel regions having a photoelectric conversion function are two-dimensionally arranged, and in each pixel region, an image signal and distance information can be acquired. The solid-state imaging device described in Patent Document 1 arranges pixels having a distance measuring function in some or all of the pixels of the imaging device, and detects the subject distance based on the phase difference detected on the imaging surface (imaging surface phase difference method).

[0003] That is, a positional shift is calculated based on the correlation relationship between two image signals based on images generated by light beams passing through different pupil regions of the imaging optical system included in the imaging device, and a distance is acquired based on the positional shift. The correlation relationship between the two image signals is evaluated using a technique such as a region-based matching technique that cuts out image signals included in a predetermined collation region from each image signal and evaluates the correlation relationship.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when the contrast change of the subject included in the image signal is small, or when the amount of noise included in the image signal is large, etc., misevaluation of the correlation relationship may occur due to the subject or shooting conditions. When the occurrence of misevaluation of the correlation relationship is included more than a certain level, the amount of positional shift between the two calculated image signals has an error, and the accuracy of the acquired distance may decrease.

[0006] The present invention aims to provide an image processing apparatus capable of reducing the effects of distance variations. [Means for solving the problem]

[0007] An image processing apparatus according to one aspect of the present invention is: A region division means for dividing image information into multiple divided regions extending in a predetermined direction, Boundary detection means for detecting boundaries based on color information for images within each of the divided regions, Integrated distance equivalent information determination means that, based on the distance equivalent information of each pixel of the image information, integrates the distance equivalent information for each sub-region separated by the boundary within the divided region to determine integrated distance equivalent information indicating the respective distance of each sub-region, A classification means that performs a predetermined classification of the sub-regions in accordance with the integrated distance equivalent information of the sub-regions, It is characterized by having the following features. [Effects of the Invention]

[0008] According to the present invention, an image processing apparatus capable of reducing the effects of distance variations can be realized. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram showing an example of the configuration of a vehicle 100 according to the first embodiment. [Figure 2] This is a block diagram showing an example of the internal configuration of a vehicle 100 according to the first embodiment. [Figure 3] This is a functional block diagram showing an example configuration of the route generation device 150 according to the first embodiment. [Figure 4] (A) and (B) are schematic diagrams showing examples of the configuration of an image sensor according to the first embodiment. [Figure 5] (A) to (D) are schematic diagrams illustrating the relationship between subject distance and incident light in the image plane phase-detection imaging system. [Figure 6] (A) and (B) are flowcharts showing examples of processing performed by the image processing unit 310 according to the first embodiment. [Figure 7] (A) and (B) are flowcharts showing examples of processes executed by the boundary processing unit 321 and the object detection unit 323 according to the first embodiment. [Figure 8] (A) to (E) are schematic diagrams showing examples of images and information in an example of a process performed by the object information generation unit 320 according to the first embodiment. [Figure 9] It is a flowchart showing a specific example of the object boundary candidate determination process in step S7011 of FIG. 7(B) performed by the boundary processing unit 321. [Figure 10] It is a flowchart showing a detailed example of the distance equivalent information integration process in step 7012 of FIG. 7(B) by the distance information generation unit 322. [Figure 11] It is a flowchart showing a detailed example of the object detection process in step 7012 of FIG. 7(B) by the object detection unit 323. [Figure 12] It is a flowchart showing an example of the generation process of distance information for each object performed by the distance information generation unit 322. [Figure 13] It is a flowchart showing an example of the path generation process executed by the path generation unit 330 according to the first embodiment. [Figure 14] It is a flowchart showing an example of the object distance information generation process performed by the distance information generation unit 322 in Modification 2. [Figure 15] It is a schematic diagram for explaining an example of the temporal change of object distance information of an object with the same identification number as the Nth object.

Embodiments for Carrying Out the Invention

[0010] [ Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each figure, the same members or elements are denoted by the same reference numerals, and overlapping descriptions are omitted or simplified. In the following description, an example of a path generation device (electronic device) equipped with a photographing device will be used for explanation, but the present invention is not limited thereto.

[0011] <First Embodiment> FIG. 1 is a schematic diagram showing a configuration example of vehicle 100 according to the first embodiment. Vehicle 100 as a moving body includes an imaging device 110, a radar device 120, a path generation ECU 130, a vehicle control ECU 140, and a measuring instrument group 160. Vehicle 100 further includes a drive unit 170 as drive control means for driving vehicle 100, a memory 190, etc. The drive unit 170, the memory 190, etc. will be described using FIG. 2.

[0012] The imaging device 110 and the path generation ECU 130, etc. constitute a path generation device 150 as an image processing device. Driver 101 can board vehicle 100, and when driving, driver 101 boards facing the front (traveling direction) of vehicle 100. Driver 101 can control the operation of the vehicle by operating operation members such as the steering wheel of vehicle 100, the accelerator pedal, and the brake pedal. Note that vehicle 100 may have an automatic driving function or may be remotely controllable from the outside.

[0013] The imaging device 110 is arranged to photograph the front (normal traveling direction) of vehicle 100. As shown in FIG. 1, the imaging device 110 is arranged inside vehicle 100 near the upper end of, for example, the front glass of vehicle 100, and photographs an area within a predetermined angular range (hereinafter, the photographing angle of view) facing the front of vehicle 100.

[0014] Note that the imaging device 110 may include an imaging device arranged to photograph the rear (opposite to the normal traveling direction, reverse direction) of vehicle 100, or may include an imaging device arranged to photograph the side. That is, a plurality of imaging devices 110 may be arranged on vehicle 100.

[0015] FIG. 2 is a block diagram showing a configuration example inside vehicle 100 according to the first embodiment. The imaging device 110 photographs the environment around the vehicle (surrounding environment) including the road (traveling road) on which vehicle 100 travels, and detects an object included within the range of the photographing angle of view of the imaging device 110.

[0016] Furthermore, the imaging device 110 acquires information about the detected object (external information) and information about the distance to the detected object (object distance information), and outputs them to the route generation ECU 130. The object distance information can be any information that can be converted into the distance from a predetermined position on the vehicle 100 to the object using a predetermined reference table or predetermined conversion coefficients or conversion formulas. For example, object distance information in which the distance is assigned to a predetermined integer value may be output sequentially to the route generation ECU 130.

[0017] The imaging device 110 includes, for example, a CMOS image sensor in which multiple pixel regions having photoelectric conversion functions are arranged in two dimensions, and is configured to acquire the distance of an object using an imaging plane phase difference method (also called an imaging plane phase difference detection method or imaging plane phase difference distance measurement method). The acquisition of object distance information using the imaging plane phase difference method will be described later.

[0018] The radar device 120 is a millimeter-wave radar device that uses electromagnetic waves with wavelengths ranging from the millimeter-wave band to the submillimeter-wave band, for example. It is a detection device that detects objects by irradiating them with electromagnetic waves and receiving the reflected waves. The radar device 120 functions as a fourth distance information acquisition means that acquires distance information (fourth distance information) indicating the distance to an object in the direction of electromagnetic wave transmission, based on the time from irradiation with electromagnetic waves to receiving the reflected waves and the received intensity of the reflected waves. The radar device 120 outputs the distance information to the path generation ECU 130.

[0019] In this embodiment, a vehicle 100 is equipped with multiple radar devices 120, for example, radar devices 120 are mounted on the left and right sides of the front of the vehicle 100, and on the left and right sides of the rear of the vehicle 100. Each radar device 120 irradiates electromagnetic waves within a predetermined angular range, and based on the time from the transmission of the electromagnetic waves to the reception of the reflected waves and the received intensity of the reflected waves, it measures the distance from each radar device 120 and generates distance information to the object. In addition to the distance from the radar device 120, the distance information may also include information on the received intensity of the reflected waves and the relative velocity of the object.

[0020] The measuring instrument group 160 includes, for example, a speed measuring instrument 161, a steering angle measuring instrument 162, and an angular velocity measuring instrument 163, and acquires vehicle information related to the vehicle's driving state, such as travel speed, steering angle, and angular velocity. The speed measuring instrument 161 is a measuring instrument that detects the travel speed of the vehicle 100. The steering angle measuring instrument 162 is a measuring instrument that detects the steering angle of the vehicle 100.

[0021] The angular velocity measuring instrument 163 is a measuring instrument that detects the angular velocity of the vehicle 100 in the turning direction. Each measuring instrument outputs a measurement signal corresponding to the measured parameter as vehicle information to the route generation ECU 130.

[0022] The route generation ECU 130 has a built-in CPU and generates the vehicle 100's trajectory and route information related to the trajectory based on measurement signals, external information, object distance information, and distance information. The route generation ECU 130 outputs this trajectory and route information to the vehicle control ECU 140. The data processed and the computer programs executed by the route generation ECU 130 are stored in memory 180.

[0023] Here, the driving trajectory is information indicating the path (route) that vehicle 100 travels. The route information is information (including road information, etc.) regarding the route that vehicle 100 travels.

[0024] The vehicle control ECU 140 has a built-in CPU and controls the drive unit 170 so that the vehicle 100 travels along the route corresponding to the route information, based on route information and vehicle information acquired from the measuring instrument group 160. The data processed and the computer programs executed by the vehicle control ECU 140 are stored in the memory 190.

[0025] The drive unit 170 is a drive component for driving a vehicle and includes, for example, a power unit (not shown) such as an engine or motor that generates energy to rotate the tires, and a steering unit that controls the direction of travel of the vehicle. The drive unit 170 also includes a gearbox for rotating the tires using the energy generated by the power unit, a gear control unit that controls the configuration within the gearbox, and a brake unit that performs braking.

[0026] The vehicle control ECU 140 controls the drive unit 170 to adjust the drive amount, braking amount, steering amount, etc. of the vehicle 100 so that the vehicle travels along the path corresponding to the path information. Specifically, the vehicle control ECU 140 controls the brakes, steering, gear configuration, etc. to make the vehicle 100 operate to travel along the path.

[0027] Furthermore, the route generation ECU 130 and the vehicle control ECU 140 may share a common CPU and common memory. HMI240 is an abbreviation for Human Machine Interface, which is used to transmit information to the driver 101.

[0028] The HMI240 includes a display that is visible when the driver 101 is in the driving position, and a display control device that generates information to be displayed on the display. The HMI240 also includes a device that outputs sound (speaker system) and a voice control device that generates voice data.

[0029] The display control unit within the HMI240 displays navigation information on the display based on the route information generated by the route generation ECU130. The voice control unit within the HMI240 also generates voice data to notify the driver 101 of the route information based on the route information, and outputs this data through the speaker system. This voice data may include, for example, data to notify the driver that an intersection to turn is approaching.

[0030] Figure 3 is a functional block diagram showing an example configuration of the route generation device 150 according to the first embodiment. Note that some of the functional blocks shown in Figure 3 are realized by having the computer included in the route generation device 150 execute a computer program stored in memory, which serves as a storage medium.

[0031] However, some or all of these may be implemented in hardware. Hardware options include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, the functional blocks shown in Figure 3 do not necessarily have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths.

[0032] In Figure 3, the path generation device 150, which functions as an image processing device, comprises an imaging device 110 and a path generation ECU 130. The imaging device 110 includes an imaging optical system 301, an image sensor 302, an image processing unit 310, an object information generation unit 320, and a memory 340. In this embodiment, the imaging optical system 301, the image sensor 302, the image processing unit 310, and the object information generation unit 320 are arranged inside the housing (not shown) of the imaging device 110. Here, the image sensor 302 functions as an imaging means that captures an image of a subject and generates image information.

[0033] The imaging optical system 301 is the imaging lens of the imaging device 110 and has the function of forming an image (optical image) of the subject on the image sensor 302. The imaging optical system 301 is composed of multiple lens groups and has an exit pupil at a predetermined distance from the image sensor 302.

[0034] The image sensor 302 is composed of a CMOS (complementary metal-oxide-semiconductor) image sensor and a CCD (charge-coupled device) image sensor, and has a distance measurement function using an image plane phase difference distance measurement method. The image sensor 302 has multiple pixel regions with photoelectric conversion functions arranged in a two-dimensional manner.

[0035] Each pixel region has, for example, two photoelectric conversion units (a first photoelectric conversion unit and a second photoelectric conversion unit) that are separated in the row direction. In front of each pixel region, for example, one color filter of R, G, or B and a microlens are placed.

[0036] The image sensor 302 converts the subject image formed on the image sensor 302 via the imaging optical system 301 into an image signal based on the subject image, and outputs it to the image processing unit 310. The image signal is a signal consisting of the output values ​​of each photoelectric conversion unit in each pixel region.

[0037] The image sensor 302 outputs a first image signal based on the signal output from the first photoelectric conversion unit and a second image signal based on the signal output from the second photoelectric conversion unit separately. Alternatively, it outputs an added signal, which is the sum of the first and second image signals, and the first image signal separately to the image processing unit 310.

[0038] The image processing unit 310 generates image data containing information for each of the red, green, and blue colors for each pixel, and distance image data indicating distance information for each pixel, based on the image signal supplied from the image sensor 302. The image processing unit 310 includes a development unit 311 that generates image data based on the image signal supplied from the image sensor 302, and a distance image generation unit 312 that generates distance image data based on the image signal supplied from the image sensor 302. The distance image generation unit 312 functions as a distance equivalent information measuring means that detects distance equivalent information for each pixel of the image information.

[0039] The processes performed by these units will be described later. The image processing unit 310 outputs image data from the development unit 311 and distance image data from the distance image generation unit 312 to the object information generation unit 320.

[0040] The object information generation unit 320 has a boundary processing unit 321 for detecting the boundaries of multiple objects in an image. The boundary processing unit 321 divides the image data from the development unit 311 into strip-shaped rectangular regions (divided regions) in the vertical direction of the screen, and further detects boundaries based on color difference signals (or color signals) for the image signals within each rectangular region. When detecting boundaries, changes in brightness information may also be taken into consideration. Here, the boundary processing unit 321 functions as a boundary detection means that divides the image into multiple divided regions extending in a predetermined direction and detects boundaries based on color information for the image within each divided region.

[0041] In the object information generation unit 320, boundaries are detected for the image signal within each rectangular region based on color difference signals, thereby classifying (grouping) the sub-regions separated by the boundaries within the divided region into three groups, for example, the sky, roads, and other objects. The reason why the boundary processing unit detects boundaries based on color difference signals for each rectangular strip-shaped region as described above is to make boundary detection easier. In this embodiment, the predetermined direction in which the rectangular strip-shaped regions (divided regions) extend is set to the vertical direction of the screen, but it may also be the horizontal direction of the screen.

[0042] The object information generation unit 320 includes a distance information generation unit 322 that determines and assigns distance information to each of the above groups. Based on boundary information from the boundary processing unit 321 and distance image data (distance equivalent information) from the distance image generation unit 312, the distance information generation unit 322 determines and assigns object distance information (integrated distance equivalent information) that indicates the distance to each of, for example, three sub-regions included in the acquired image. Here, the distance information generation unit 322 functions as an integrated distance equivalent information determination means that determines integrated distance equivalent information for each sub-region separated by the boundary within the divided region, based on the distance equivalent information of each pixel of the image information.

[0043] In this embodiment, boundaries are made easier to detect by detecting them based on color difference signals for each rectangular strip-shaped region. Furthermore, since integrated distance equivalent information is determined for, for example, three sub-regions separated by the boundaries, the influence of distance errors (variations) that occur in the distance image generation unit 312 can be suppressed. In this embodiment, boundaries are detected based on color difference signals, but boundaries may also be detected based on color signals. Alternatively, the target boundaries may be detected by referring not only to color difference signals or color signals but also to luminance signals.

[0044] In particular, when acquiring the distance of an object using an image sensor with an image plane phase difference method, as in this embodiment, distance errors (variations) may occur for distant objects due to the relatively short baseline length, but this effect can be significantly suppressed.

[0045] The object information generation unit 320 includes an object detection unit 323 that performs image recognition of objects. The object detection unit 323 integrates image signals of strip-shaped rectangular regions whose boundaries are set by the boundary processing unit 321 in the horizontal direction of the image, and based on the integrated image, it performs image recognition of objects included in the acquired image, such as the sky, roads, and other objects. Here, the object detection unit 323 functions as an object detection means that performs object detection by integrating images of multiple divided regions, including classified sub-regions.

[0046] For example, the group at the bottom of the screen is determined to be a road. Here, the object detection unit 323 functions as a classification means that classifies the sub-regions, for which integrated distance equivalent information has been determined, according to the integrated distance equivalent information of the sub-regions. Furthermore, as described above, the object detection unit 323 in this embodiment classifies the sub-regions into one of three categories: sky, roads, or other.

[0047] Furthermore, when the object detection unit 323 performs image recognition of the sky, roads, and other objects, it refers to the distance information assigned to each separated region by the distance information generation unit 322. In other words, by recognizing objects with a distance difference of more than a predetermined amount as different objects, misrecognition can be reduced. In addition, the group at the top of the screen that is at infinity is more easily recognized as the sky.

[0048] By distinguishing between the sky and the road in this way, object recognition can be focused on objects other than the sky and the road. The object detection unit 323 can accurately generate external information that shows information about the detected sky, road, and other objects. Here, the external information is information that shows the position of the detected object in the image, its size such as width and height, and its area. The external information also includes information about the attributes and identification number of the detected object.

[0049] Furthermore, object distance information is linked to information regarding the identification number of the object included in the external environment information. The object information generation unit 320 outputs the external environment information from the object detection unit 323 and the object distance information from the distance information generation unit 322 to the route generation ECU 130.

[0050] Furthermore, the image processing unit 310 and the object information generation unit 320 may be composed of one or more processors in the imaging device 110. Alternatively, one or more processors may execute a program read from the memory 340 to realize the functions of the image processing unit 310 and the object information generation unit 320.

[0051] The route generation ECU 130 includes a route generation unit 330. The route generation unit 330 generates route information based on external information, object distance information, and distance information acquired from the radar device 120. Next, the structure and control of each block within the route generation device 150 will be described in detail.

[0052] Figures 4(A) and 4(B) are schematic diagrams showing an example configuration of the image sensor 302 according to the first embodiment. Figure 4(A) is a top view of the image sensor 302 as seen from the direction of light incidence. The image sensor 302 is composed of multiple 2x2 pixel groups 410 arranged in a matrix. The pixel group 410 includes green pixels G1 and G2 for detecting green light, red pixels R for detecting red light, and blue pixels B for detecting blue light. In the pixel group 410, green pixels G1 and G2 are arranged diagonally. Each pixel also has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412.

[0053] Figure 4(B) is a cross-sectional view of the pixel group 410 in the I-I' section of Figure 4(A). Each pixel consists of a light guide layer 414 and a light receiving layer 415. The light guide layer 414 has a microlens 413 for efficiently guiding the light beam incident on the pixel to the light receiving layer 415, a color filter (not shown) that allows light in a wavelength band corresponding to the color of light detected by each pixel to pass through, and wiring for image readout and pixel driving.

[0054] The light-receiving layer 415 is a photoelectric conversion unit that converts light incident through the light-guide layer 414 into electrical signals and outputs them as electrical signals. The light-receiving layer 415 has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412.

[0055] Figures 5(A) to 5(D) are schematic diagrams illustrating the relationship between subject distance and incident light in the image plane phase-difference imaging method. Figure 5(A) is a schematic diagram showing the exit pupil 501 of the imaging optical system 301, the green pixel G1 of the image sensor 302, and the light incident on each photoelectric conversion part of the green pixel G1. The image sensor 302 has multiple pixels, but for simplicity, only one green pixel G1 will be described.

[0056] The microlens 413 of the green pixel G1 is arranged such that the exit pupil 501 and the light-receiving layer 415 are optically conjugate. As a result, the light beam that passes through the first pupil region 510, which is a partial pupil region of the exit pupil 501, is incident on the first photoelectric conversion unit 411. Similarly, the light beam that passes through the second pupil region 520, which is a partial pupil region, is incident on the second photoelectric conversion unit 412.

[0057] The first photoelectric conversion unit 411 of each pixel converts the received light beam into electricity and outputs a signal. A first image signal is generated from the signals output from the multiple first photoelectric conversion units 411 contained in the image sensor 302. The first image signal shows the intensity distribution of the image (referred to as image A) formed on the image sensor 302 by the light beam that mainly passed through the first pupil region 510.

[0058] The second photoelectric conversion unit 412 of each pixel converts the received light beam into electricity and outputs a signal. A second image signal is generated from the signals output from the multiple second photoelectric conversion units 412 contained in the image sensor 302. The second image signal shows the intensity distribution of the image (referred to as the B image) formed on the image sensor 302 by the light beam that mainly passed through the second pupil region 520.

[0059] The relative positional shift (hereinafter referred to as the disparity amount) between the first image signal corresponding to image A and the second image signal corresponding to image B is a quantity corresponding to the defocus amount. The relationship between the disparity amount and the defocus amount will be explained using Figures 5(B), (C), and (D).

[0060] Figures 5(B), (C), and (D) are schematic diagrams showing the positional relationship between the image sensor 302 and the imaging optical system 301. In the figures, 511 indicates the first light beam passing through the first pupil region 510, and 521 indicates the light beam passing through the second pupil region 520.

[0061] Figure 5(B) shows the state when the image is in focus, with the first light beam 511 and the second light beam 521 converging on the image sensor 302. At this time, the amount of disparity (positional shift) between the first image signal corresponding to image A formed by the first light beam 511 and the second image signal corresponding to image B formed by the second light beam 521 is 0.

[0062] Figure 5(C) shows a state where the image is defocused in the negative z-axis direction. At this time, the disparity between the first image signal formed by the first light beam 511 and the second image signal formed by the second light beam 521 is not zero, but has a negative value.

[0063] Figure 5(D) shows the state where the image is defocused in the positive z-axis direction. At this time, the disparity between the first image signal formed by the first light beam 511 and the second image signal formed by the second light beam 521 is not zero, but has a positive value.

[0064] A comparison of Figures 5(C) and (D) shows that the direction in which parallax occurs reverses depending on whether the amount of defocus is positive or negative. Furthermore, the geometric relationship shows that the amount of parallax corresponds to the amount of defocus. Therefore, as will be described later, the amount of parallax between the first image signal and the second image signal can be detected using a region-based matching method, and the amount of parallax can be converted into the amount of defocus via a predetermined conversion coefficient. Moreover, by using the imaging formula of the imaging optical system 301, the amount of defocus on the image side can be converted into the distance to the object.

[0065] Furthermore, as described above, the image sensor 302 may output the summation signal (composite signal) of the first image signal and the second image signal, and the first image signal separately to the image processing unit 310. In this case, the image processing unit 310 can generate the second image signal by the difference between the summation signal (composite signal) and the first image signal, and acquire the first image signal and the second image signal, respectively.

[0066] Next, the processing performed by the image processing unit 310 will be described. Figures 6(A) and 6(B) are flowcharts showing examples of processing performed by the image processing unit 310 according to the first embodiment. Note that the CPU, which acts as a computer included in the path generation device 150, executes a computer program stored in memory, thereby performing the operation of each step in the flowcharts in Figures 6(A) and 6(B).

[0067] Figure 6(A) is a flowchart showing the operation of the development process in which the development unit 311 of the image processing unit 310 generates image data from the image signals (first image signal and second image signal). The development process is performed in response to the reception of an image signal from the image sensor 302.

[0068] In step S601, the CPU generates a composite image signal by combining the first image signal and the second image signal input from the image sensor 302. Alternatively, as described above, the summation signal may be read out at the stage of reading from the image sensor 302. By combining (adding) the first image signal and the second image signal, an image signal based on the image formed by the light beam that passed through the entire area of ​​the exit pupil 501 can be obtained.

[0069] When the horizontal pixel coordinate of the image sensor 302 is x and the vertical pixel coordinate is y, the composite image signal Im(x,y) of the pixel (x,y) can be expressed by the following equation 1 using the first image signal Im1(x,y) and the second image signal Im2(x,y).

[0070]

number

[0071] In step S602, the development unit 311 performs a correction process for defective pixels in the composite image signal. Defective pixels are pixels in the image sensor 302 that cannot produce a normal signal output. The coordinates of the defective pixels are stored in memory beforehand, and the development unit 311 obtains information indicating the coordinates of the defective pixels in the image sensor 302 from the memory.

[0072] The developing unit 311 generates the composite image signal of the defective pixel using a median filter that replaces the composite image signal of the pixels surrounding the defective pixel with the median value of the composite image signals of the pixels surrounding the defective pixel. Alternatively, as a method for correcting the composite image signal of the defective pixel, the signal value of the defective pixel may be generated by interpolating using pre-prepared coordinate information of the defective pixel and the signal values ​​of the pixels surrounding the defective pixel.

[0073] In step S603, the development unit 311 applies a light intensity correction process to the composite image signal to compensate for the reduction in light intensity around the field of view caused by the imaging optical system 301. The light intensity reduction characteristics (relative light intensity ratio) around the field of view caused by the imaging optical system 301 are measured in advance and stored in memory. As a method of light intensity correction, the relative light intensity ratio between fields of view stored in advance can be read from memory, and the composite image signal can be corrected by multiplying it by a gain such that the light intensity ratio becomes constant. For example, the development unit 311 performs light intensity correction by multiplying the composite image signal of each pixel by a gain that has the characteristic of increasing from the central pixel to the peripheral pixels of the image sensor 302.

[0074] In step S604, the developing unit 311 performs noise reduction processing on the composite image signal. As a method for reducing noise, noise reduction using a Gaussian filter can be used.

[0075] In step S605, the developing unit 311 performs demosaicing on the composite image signal to acquire red (R), green (G), and blue (B) color signals for each pixel, and uses these to generate image data containing brightness information for each pixel. As a demosaicing method, a technique can be used in which the color information for each pixel is interpolated using linear interpolation for each color channel.

[0076] In step S606, the developing unit 311 performs gradation correction (gamma correction processing) using a predetermined gamma value. The image data Idc(x,y) of the pixel (x,y) after gradation correction is expressed by the following formula 2 using the image data Id(x,y) of the pixel (x,y) before gradation correction and the gamma value γ.

[0077]

number

[0078] The gamma value γ can be a pre-defined value. The gamma value γ may also be determined according to the pixel position. For example, the gamma value γ may be changed for each region obtained by dividing the effective area of ​​the image sensor 302 into a predetermined number of divisions.

[0079] In step S607, the development unit 311 performs a color space conversion process to convert the color space of the image data from the RGB color space to the YUV color space. The development unit 311 uses predetermined coefficients and a color space conversion formula (equation 3) to convert the image data corresponding to the luminance of each color (red, green, and blue) into luminance values ​​and color difference values, thereby converting the color space of the image data from the RGB color space to the YUV color space.

[0080]

number

[0081] In Equation 3, IdcR(x,y) represents the red image data value of pixel (x,y) after tone correction. IdcG(x,y) represents the green image data value of pixel (x,y) after tone correction. IdcB(x,y) represents the blue image data value of pixel (x,y) after tone correction. Y(x,y) represents the luminance value of pixel (x,y) obtained by color space conversion. U(x,y) represents the difference (chrominance value) between the luminance value and the blue component of pixel (x,y) obtained by color space conversion.

[0082] V(x,y) represents the difference (chrominance value) between the luminance value and the red component of the pixel (x,y) obtained by color space conversion. The coefficient (ry,gy,gy) is used to find Y(x,y), and the coefficients (ru,gu,gu) and (rv,gv,bv) are used to calculate the chrominance value, respectively.

[0083] In step S608, the developing unit 311 performs a correction (distortion correction) on the converted image data to suppress the effect of distortion aberration caused by the optical characteristics of the imaging optical system 301. The distortion aberration correction process is performed by geometrically transforming the image data to correct the distortion rate of the imaging optical system 301. The geometrical transformation is performed using a polynomial that generates the pixel positions before correction from the correct pixel positions without distortion aberration. If the pixel positions before correction are decimal numbers, they may be rounded to the nearest neighbor pixel, or linear interpolation may be used.

[0084] In step S609, the developing unit 311 outputs the image data to which distortion correction processing has been applied to the object information generation unit 320. This completes the developing process performed by the developing unit 311. The flow shown in Figure 6(A) is repeated periodically until the user issues a termination instruction (not shown).

[0085] Furthermore, if the boundary processing unit 321 and the object detection unit 323 can perform boundary detection and external recognition processing based on image data obtained from the image sensor 302 before various correction processing, the development unit 311 does not need to perform all the processing shown in Figure 6(A). For example, if the boundary processing unit 321 and the object detection unit 323 can perform boundary detection and detect objects within the shooting angle range based on image data to which the distortion correction processing in step S608 has not been applied, the processing in step S608 may be omitted from the development processing in Figure 6(A).

[0086] Figure 6(B) is a flowchart showing the operation of the distance image generation process performed by the distance image generation unit 312. Here, distance image data is data to which distance information corresponding to the distance from the imaging device 110 to the subject is associated with each pixel. The distance information may be a distance value D, or a defocus amount ΔL or disparity amount d used to calculate the distance value, but in this embodiment, the distance image data will be described as data to which a distance value D is associated with each pixel.

[0087] In step S611, the distance image generation unit 312 generates first and second luminance image signals from the input image signal. That is, the distance image generation unit 312 generates a first luminance image signal using the first image signal corresponding to image A, and generates a second luminance image signal using the second image signal corresponding to image B. At that time, the distance image generation unit 312 multiplies the image signal values ​​of the red, green, and blue pixels of each pixel group 410 by a predetermined coefficient, and then synthesizes them to generate a luminance image signal. Alternatively, the distance image generation unit 312 may generate a luminance image signal by performing demosaicing using linear interpolation and then multiplying the red, green, and blue pixel signals by a predetermined coefficient and synthesizing them.

[0088] In step S612, the distance image generation unit 312 corrects the light intensity balance between the first luminance image signal and the second luminance image signal. The light intensity balance correction is performed by multiplying at least one of the first luminance image signal and the second luminance image signal by a predetermined correction coefficient. The correction coefficient is calculated in advance by measuring the luminance ratio between the obtained first luminance image signal and the second luminance image signal after uniform illumination is applied after the position adjustment of the imaging optical system 301 and the image sensor 302, and the coefficient is stored in the memory 340 so that the luminance ratio remains constant.

[0089] The distance image generation unit 312 multiplies the correction coefficient read from memory by at least one of the first luminance image signal and the second luminance image signal to generate a first image signal and a second image signal to which light intensity balance correction has been applied.

[0090] In step S613, the distance image generation unit 312 performs noise reduction processing on the first luminance image signal and the second luminance image signal to which light intensity balance correction has been applied. The distance image generation unit 312 performs noise reduction processing by applying a low-pass filter that reduces high spatial frequency bands to each luminance image signal. Alternatively, the distance image generation unit 312 may use a band-pass filter that transmits a predetermined spatial frequency band. In this case, the effect of reducing the influence of the correction error of the light intensity balance correction performed in step S612 can be obtained.

[0091] In step S614, the distance image generation unit 312 calculates the disparity amount, which is the relative positional shift between the first luminance image signal and the second luminance image signal. The distance image generation unit 312 sets a point of interest in the first luminance image corresponding to the first luminance image signal and sets a matching region centered on the point of interest. Next, the distance image generation unit 312 sets a reference point in the second luminance image corresponding to the second luminance image signal and sets a reference region centered on the reference point.

[0092] The distance image generation unit 312 calculates the correlation between a first luminance image contained within the matching region and a second luminance image contained within the reference region while sequentially moving the reference point, and identifies the reference point with the highest correlation as the corresponding point. The distance image generation unit 312 defines the amount of relative positional shift between the point of interest and the corresponding point as the amount of disparity at the point of interest. By calculating the amount of disparity while sequentially moving the point of interest, the distance image generation unit 312 can calculate the amount of disparity at multiple pixel positions. In this manner, the distance image generation unit 312 identifies a value indicating the disparity value for each pixel and generates disparity image data, which is data indicating the disparity distribution.

[0093] Furthermore, known methods can be used as the method for calculating the correlation used by the distance image generation unit 312 to determine the amount of disparity. For example, the distance image generation unit 312 can use a method called NCC (Normalized Cross-Correlation) which evaluates the normalized cross-correlation between luminance images. Alternatively, the distance image generation unit 312 may use a method that evaluates the degree of difference as the correlation. For example, the distance image generation unit 312 can use SAD (Sum of Absolute Difference), which evaluates the sum of the absolute differences between luminance images, or SSD (Sum of Squared Difference), which evaluates the sum of the squares of the differences.

[0094] In step S615, the distance image generation unit 312 converts the disparity amount of each pixel in the disparity image data into a defocus amount to obtain the defocus amount of each pixel. Based on the disparity amount of each pixel in the disparity image data, the distance image generation unit 312 generates defocus image data that shows the defocus amount of each pixel.

[0095] The distance image generation unit 312 uses the disparity amount d(x,y) of the pixel (x,y) in the disparity image data and the conversion coefficient K(x,y) to calculate the defocus amount ΔL(x,y) of the pixel (x,y) from the following equation 4. Note that the imaging optical system 301 loses a portion of the first light beam 511 and the second light beam 521 in the peripheral field of view due to vignetting. Therefore, the conversion coefficient K is a value that depends on the field of view (pixel position).

[0096]

number

[0097] If the imaging optical system 301 has a characteristic of field curvature in which the focal position changes between the central and peripheral fields of view, then, if the amount of field curvature is Cf, the parallax amount d(x,y) can be converted to the defocus amount ΔL(x,y) using the following equation 5. After aligning the imaging optical system 301 and the image sensor 302, the relationship between the amount of parallax and the distance to the object can be obtained by chart imaging, thereby obtaining the conversion coefficient K and the amount of field curvature Cf. In this case, the amount of field curvature Cf depends on the field of view and is given as a function of the pixel position.

[0098]

number

[0099] In step S616, the distance image generation unit 312 converts the defocus amount ΔL(x,y) of pixel (x,y) to the distance value D(x,y) to the object at pixel (x,y), and generates distance image data. The distance value D to the object can be calculated by converting the defocus amount ΔL using the imaging relationship of the imaging optical system 301. When the focal length of the imaging optical system 301 is f and the distance from the principal point on the image side to the image sensor 302 is Ipp, the defocus amount ΔL(x,y) can be converted to the distance value D(x,y) to the object using the imaging formula in equation 6 below.

[0100]

number

[0101] In this example, the focal length f and the distance Ipp from the principal point on the image side to the image sensor 302 are assumed to be constant values ​​regardless of the field of view, but this is not limited to this. If the imaging magnification of the imaging optical system 301 changes significantly with each field of view, at least one of the focal length f or the distance Ipp from the principal point on the image side to the image sensor 302 may be set to a value that changes with each field of view.

[0102] In step S617, the distance image generation unit 312 outputs the distance image data generated as described above to the object information generation unit 320. Here, through the processing in steps S611 to S617, the distance image generation unit 312 measures distance equivalent information for each pixel using a phase difference distance measurement method based on signals from the first photoelectric conversion unit and the second photoelectric conversion unit located within the pixels of the imaging means.

[0103] Furthermore, in order to acquire distance equivalent information, distance equivalent information may be measured using a phase difference distance measurement method based on two image signals from a stereo camera. This completes the distance image data generation process performed by the distance image generation unit 312. The flow shown in Figure 6(B) will be repeated periodically until the user issues a termination instruction (not shown).

[0104] Furthermore, the parallax amount d, defocus amount ΔL, and distance value D from the principal point of the imaging optical system 301 for each pixel are values ​​that can be converted using the coefficients and conversion formulas described above. Therefore, the distance image data generated by the distance image generation unit 312 may include information representing the parallax amount d or the defocus amount ΔL for each pixel. Considering that the object information generation unit 320 calculates a representative value of the distance value D included in the object region, it is desirable to generate the distance image data based on a defocus amount whose frequency distribution is symmetrical.

[0105] As mentioned above, in the process of calculating the disparity amount in step S614, the correlation between the first luminance image and the second luminance image is used to search for corresponding points. However, if there is a lot of noise in the first image signal (for example, noise caused by optical shot noise) or if the change in the signal value of the luminance image signal included in the matching area is small, it may not be possible to correctly evaluate the degree of correlation. In such cases, a disparity amount with a large error may be calculated compared to the correct disparity amount. If the error in the disparity amount is large, the error in the distance value D generated in step S616 will also be large.

[0106] Therefore, the distance image generation process performed by the distance image generation unit 312 may include a reliability calculation process for calculating the reliability of the disparity amount (disparity reliability). Disparity reliability is an indicator of how much error is contained in the calculated disparity amount. For example, the ratio of the standard deviation to the mean value of the signal values ​​included in the matching area can be evaluated as the disparity reliability. When the change in signal values ​​within the matching area (so-called contrast) is large, the standard deviation becomes large. When the amount of light incident on the pixel is large, the mean value becomes large. When the amount of light incident on the pixel is large, there is a lot of light shot noise. That is, the mean value has a positive correlation with the amount of noise.

[0107] The ratio of the mean to the standard deviation (standard deviation / mean) corresponds to the ratio of the contrast magnitude to the noise level. If the contrast is sufficiently large relative to the noise level, it can be estimated that the error in calculating the parallax amount is small. In other words, the higher the parallax confidence, the smaller the error in the calculated parallax amount, and the more accurate the parallax amount is considered to be.

[0108] Furthermore, the reliability may be lowered for pixels with saturated pixel signals and their surrounding pixels. For example, even if there is contrast in the vicinity of a saturated pixel, the texture in the reference image and the reference image may not be detected correctly, and the parallax may not be calculated correctly. Therefore, if the pixel signal is above a predetermined value (saturated), the parallax reliability may be lowered for the vicinity of that pixel.

[0109] Therefore, in step S614, the parallax confidence level is calculated at each point of interest, and confidence data representing the certainty of the distance value for each pixel constituting the distance image data can be generated. The distance image generation unit 312 can output the confidence data to the object information generation unit 320.

[0110] Next, the process by which the object information generation unit 320 generates external information and object distance information based on image data and distance image data will be explained with reference to Figures 7(A) and (B).

[0111] Figures 7(A) and 7(B) are flowcharts illustrating examples of processing performed by the boundary processing unit 321 and object detection unit 323 according to the first embodiment. Figures 8(A) to 8(E) are schematic diagrams showing examples of images and information in an example of processing performed by the object information generation unit 320 according to the first embodiment. The CPU, acting as a computer included in the route generation device 150, executes a computer program stored in memory to perform each step in the flowcharts of Figures 7(A) and 7(B). The steps in the flowcharts of Figures 9 to 14, described later, are also performed by the CPU, acting as a computer included in the route generation device 150, executing a computer program stored in memory.

[0112] Figure 7(A) is a flowchart showing an example of a process in which the boundary processing unit 321 generates boundaries within a rectangular strip-shaped region and classifies them, for example, into sky, road, and other objects. In the boundary and object detection process of step S701, the boundary processing unit 321 divides the image data from the development unit 311 into rectangular strip-shaped regions, for example, in the vertical direction of the screen.

[0113] Figure 8(A) illustrates an example in which an image 810, based on image data acquired by the imaging device 110 and input to the object information generation unit 320, is divided into strip-shaped rectangular regions in the vertical direction of the screen by the boundary processing unit 321. In Figure 8(A), the image 810 includes a person 801, a vehicle 802, a sign 803, a road 804, and a lane 805.

[0114] As shown in Figure 8(A), in this embodiment, the color boundaries are detected when the image 810 is divided into strip-shaped rectangular regions in the vertical direction of the screen, but the width of the strip-shaped rectangular regions is not limited to the example shown in Figure (A). Also, in this embodiment, the screens shown in Figures 8(A) to (E) are displayed, for example, on an unshown display unit inside a vehicle, but in that case, the strip-shaped rectangular regions do not need to be displayed.

[0115] The boundary processing unit 321 further detects boundaries in the image within each rectangular strip based on the color difference signal (or color signal). Figure 7(B) is a flowchart illustrating the boundary and object detection process performed by the boundary processing unit 321 in step S701.

[0116] As shown in Figure 7(B), in the boundary and object detection process in step S701, the boundary processing unit 321 first performs object boundary candidate determination processing in step S7011. Next, the distance information generation unit 322 performs distance equivalent information integration processing in step S7012. Then, in step S7013, the object detection unit 323 performs object detection processing within the rectangular strip-shaped area. The object detection processing in step S7013 is a process to classify the rectangular strip-shaped area into empty space, roads, and other objects.

[0117] Figure 9 is a flowchart showing a specific example of the object boundary candidate determination process in step S7011 of Figure 7(B) performed by the boundary processing unit 321. In step S901, the boundary processing unit 321 converts the RGB image to a Lab image. Next, in step S902, the boundary processing unit 321 calculates the amount of change in color difference in the vertical (horizontal) direction of the screen.

[0118] Specifically, for example, starting from the top edge of the screen, the color distance (color change amount) ΔC is calculated for each pixel, between two pixels that are several pixels apart in the vertical direction of the screen. Then, in step S903, it is determined whether ΔC is greater than a predetermined threshold Th. If yes, it is set as a distance boundary candidate in step S904. That is, pixels with large color changes are set as boundary candidates. In other words, the position where the color information changes beyond a predetermined threshold Th is detected as the boundary.

[0119] Furthermore, each rectangular strip (divided region) has a width in the direction intersecting the vertical direction (predetermined direction) (horizontal direction), and contains multiple pixels in the intersecting direction (horizontal direction). Therefore, the color information mentioned above refers to a representative value obtained based on the color information of multiple pixels arranged horizontally. Here, the representative value is calculated using the median or mean, etc.

[0120] On the other hand, if the answer in step S903 is No, then in step S905, it is set as a non-distance boundary pixel. In other words, pixels with a small amount of color change are set not to be boundaries.

[0121] Then, in step S906, it is determined whether the setting of distance boundary candidates or non-distance boundary candidates has been completed for all pixels in the rectangular strip area. If the answer is No, the process returns to step S902 and steps S902 to S906 are repeated.

[0122] If the answer in step S906 is Yes, then in step S907, the pixels that are candidates for distance boundary are provisionally set as distance boundaries. Next, in step S908, if the interval between those distance boundaries is less than or equal to a predetermined value (e.g., 10 pixels), the setting of the boundary for that pixel is canceled. Then, in step S909, the final distance boundaries are set, where the pixel distance between them is greater than or equal to a predetermined value. As described above, steps S901 to S909 function as boundary detection steps that divide the image information into multiple divided regions extending in a predetermined direction and detect boundaries based on color information for the image within each divided region.

[0123] Figure 10 is a flowchart showing a detailed example of the distance equivalent information integration processing in step 7012 of Figure 7(B) by the distance information generation unit 322. First, in step S1101, the distance boundary rectangle generation processing is performed. Specifically, a distance boundary rectangle with a similar color is generated, enclosed by the distance boundary and the width of the rectangular strip area.

[0124] Then, in step S1002, the distance information of each pixel within the distance boundary rectangle is weighted based on the confidence level to generate distance equivalent information for each distance boundary rectangle (hereinafter sometimes referred to as a sub-region). At this time, a confidence level representing the certainty of the distance equivalent information of each pixel may be generated based on the color information, and the integrated distance equivalent information for each sub-region may be determined based on the distance equivalent information of each pixel weighted based on the confidence level.

[0125] In this case, the amount of parallax of multiple images obtained from the image sensor is calculated based on the color information, the parallax confidence level indicating the certainty of the parallax amount of each pixel is calculated based on the color information, and the confidence level is generated based on the parallax confidence level. At that time, the parallax confidence level of pixels whose brightness value in multiple images is above a predetermined value may be calculated to be lower than that of pixels whose brightness value is below a predetermined value.

[0126] Furthermore, average height information for each distance boundary rectangle is generated by averaging the heights of pixels within that distance boundary rectangle. Alternatively, the median or mode of distance information for pixels within the distance boundary rectangle whose confidence level is above a predetermined value may be used as distance equivalent information.

[0127] Next, in step S1003, the ratio of pixels in each distance boundary rectangle whose distance information confidence level is greater than or equal to a predetermined value is calculated. Alternatively, statistical values ​​of distance equivalent information or height information (for example, the ratio of pixels below the standard deviation) are calculated for each distance boundary rectangle.

[0128] Next, in step S1004, it is determined whether the proportion of pixels with a reliability of distance information equal to or greater than a predetermined value is equal to or greater than a predetermined proportion. If the result is No, in step S1005, information indicating that the reliability of that distance boundary rectangle is low is added to that distance boundary rectangle. On the other hand, if the result in step S1004 is Yes, integrated distance equivalent information and height information are determined and added to that distance boundary rectangle.

[0129] Then, in step S1007, a map is generated for each distance boundary rectangle, which contains integrated distance equivalent information and height information. Here, steps S1001 to S1007 function as integrated distance equivalent information determination steps, which determine the integrated distance equivalent information for each sub-region separated by the boundary within the divided region, based on the distance equivalent information of each pixel in the image information.

[0130] Figure 11 is a flowchart showing a detailed example of the object detection process in step 7012 of Figure 7(B) by the object detection unit 323. In step S1101, the object boundary is searched for from the top of the screen for each rectangular strip-shaped region. Then, in step S1102, distance value processing is performed within the rectangular strip-shaped region. Specifically, it is compared with distance equivalent information within the distance boundary rectangle to determine whether it is closer than the equivalent of infinity (or a predetermined value). If the result is No, in step S1103 it is set as an empty area, and the process returns to step S1102, continuing to determine whether each distance boundary rectangle is empty, a road, or something else, starting from the top of the screen.

[0131] If the answer in step S1102 is Yes, the empty area searched from the top edge of the screen is cut off and enters another object area. Therefore, in step S1104, it is determined that the pixel is the boundary between the empty space and the object. Next, in step S1105, the object boundary is searched from the pixel at the bottom edge of the rectangular strip area. Then, in step S1106, the average height information of the distance boundary rectangle is compared with a predetermined threshold. That is, the predetermined threshold is used to determine whether the height of the distance boundary rectangle is higher than, for example, the height of a camera installed on a vehicle. Note that instead of the camera height, the median of the height information of pixels in the lower half of the screen may be used as the predetermined threshold. This is because the lower half of the screen is likely to be occupied by a road.

[0132] If the answer in step S1106 is No, the height of the distance boundary rectangle is lower than the camera position, so in step S1107 it is set as a road area, and the search continues from the bottom to the top of the screen, returning to step S1106. On the other hand, if the answer in step S1106 is Yes, then in step S1108 it is set as the boundary between the road and other objects.

[0133] Then, in step S1109, the distance boundary rectangle within each rectangular strip-shaped region is classified into three categories: road, sky, or other object. Here, steps S1101 to S1109 function as classification steps that perform a predetermined classification on sub-regions for which integrated distance equivalent information has been determined, according to the integrated distance equivalent information of the sub-regions. In this embodiment, the height of the sub-regions is also referred to in the classification.

[0134] As described above, for each rectangular strip-shaped region, the distance boundary rectangle within the rectangular strip-shaped region is classified into three categories: road, sky, or other object. Subsequently, in step S702 of Figure 7(A), the object detection unit 323 integrates the rectangular strip-shaped regions horizontally and performs object detection.

[0135] Since each rectangular strip-shaped region is pre-classified into sky, road, and other areas, when the rectangular strip-shaped regions are integrated horizontally and object detection is performed, object recognition can be focused on objects other than the sky and road, reducing the load on image recognition processing. In addition, the object detection unit 323 can accurately generate external information indicating information about the detected sky, road, and other objects.

[0136] Furthermore, when performing object detection, the object detection unit 323 may perform a process to expand or contract the size of the image data input from the image processing unit 310 to a size determined from the detection performance and processing time in the object detection process.

[0137] The object detection unit 323 generates external information based on the image data, including the position of the object in the image, its size (such as width and height), area information indicating the region, the type (attributes) of the object, and an identification number (ID number). The identification number is identification information for identifying the detected object and is not limited to a number. The object detection unit 323 detects the type of object present within the imaging angle of the imaging device 110, as well as the position and size of the object in the image, and determines whether or not the object has already been registered, and assigns an identification number.

[0138] The object detection unit 323 detects objects from the image 810 and generates external information indicating the type, identification number, and region information of each object. Figure 8(B) is a schematic diagram showing the external information of each object detected from the image 810 at each position in the image 810 on the xy coordinate plane. Note that while strip-shaped rectangular regions are displayed in Figures 8(A) to (E), these strip-shaped rectangular regions do not need to be displayed on the screen of the display unit not shown.

[0139] Furthermore, external information is generated as a table, for example, as shown in Table 1. In the external information, the area of ​​an object is defined as a rectangular frame (object frame) surrounding the object. In the external information, the area information of an object is indicated by the shape of the rectangular object frame as the coordinates of the top left (x0, y0) and bottom right (x1, y1).

[0140] [Table 1]

[0141] In step S702, the object detection unit 323 integrates the rectangular strip-shaped regions horizontally and then performs a process to detect objects contained in the image based on the image data, detecting the region corresponding to the object in the image and the type of object. The object detection unit 323 may detect multiple objects from a single image. In this case, the object detection unit 323 identifies the type and region for each of the detected objects.

[0142] The object detection unit 323 generates external information including the position and size (horizontal width, vertical height) of the area in the image where an object is detected, and the type of object. The types of objects that the object detection unit 323 can detect include, for example, vehicles (passenger cars, buses, trucks), people, animals, motorcycles, signs, etc. The object detection unit 323 detects an object and identifies the type of the detected object by comparing a predetermined outline pattern associated with the type of object with the outline of the object in the image. The types of objects that the object detection unit 323 can detect are not limited to those listed above, but from the viewpoint of processing speed, it is desirable to narrow down the number of object types to be detected according to the driving environment of the vehicle 100.

[0143] In step S703, the object detection unit 323 tracks objects for which identification numbers have already been registered. The object detection unit 323 identifies objects for which identification numbers have already been registered from among the objects detected in step S702. Objects for which identification numbers have been registered are, for example, objects that were detected in previous object detection processes and assigned identification numbers. When the object detection unit 323 detects an object for which an identification number has been registered, it associates the information about the type and region of the object obtained in step S702 with the external information corresponding to that identification number (updates the external information).

[0144] Furthermore, if it is determined that an object with a registered identification number does not exist in the image information, it is determined that the object associated with that identification number has moved outside the field of view of the imaging device 110 (lost), and tracking is suspended.

[0145] In step S704, the object detection unit 323 determines whether each of the objects detected in step S702 is a new object for which an identification number has not been registered. Then, it assigns a new identification number to the external information indicating the type and region of the object determined to be a new object, and registers it in the external information.

[0146] In step S705, the object detection unit 323 outputs the generated external information along with time information to the path generation device 150. This completes the external information generation process performed by the object detection unit 323. However, the flow shown in Figure 7(A) is repeated periodically until the user issues a termination instruction (not shown).

[0147] Figure 12 is a flowchart showing an example of the distance information generation process for each object performed by the distance information generation unit 322. Based on the external information and distance image data, the distance information generation unit 322 generates object distance information representing the distance value for each detected object. Figure 8(C) is a diagram showing an example of a distance image 820, which is generated based on the image data in Figure 8(A) and associates the distance image data with the image 810. In the distance image 820 of Figure 8(C), the distance information is shown by the intensity of the color, with darker colors indicating closer objects and lighter colors indicating farther objects.

[0148] In step S1201, the distance information generation unit 322 counts the number of objects N detected by the object detection unit 323 and calculates the total number of detected objects, Nmax.

[0149] In step S1202, the distance information generation unit 322 sets N to 1 (initialization process). The processes from step S1203 onward are executed sequentially for each object shown in the external information. Assume that the processes from step S1203 to step S1206 are executed in order of increasing identification number in the external information.

[0150] In step S1203, the distance information generation unit 322 identifies a rectangular region on the distance image 820 that corresponds to the region (object frame) on the image 810 of the Nth object included in the external information. The distance information generation unit 322 sets a frame (object frame) on the distance image 820 that shows the outline of the corresponding region.

[0151] Figure 8(D) is a schematic diagram in which frames indicating the outline of the region set in the distance image 820 are superimposed on each object detected from the image 810. As shown in Figure 8(D), the distance information generation unit 322 sets object frames 821 corresponding to the person 801, object frame 822 corresponding to the vehicle 802, and object frame 823 corresponding to the sign 803 on the distance image 820.

[0152] In step S1204, the distance information generation unit 322 generates a frequency distribution of distance information for pixels contained within the rectangular region of the distance image 820 corresponding to the Nth object. If the information associated with each pixel of the distance image data is the distance value D, the intervals of the frequency distribution are set so that the reciprocals of the distances are equally spaced.

[0153] Furthermore, if each pixel of the distance image data is associated with a defocus amount or disparity amount, it is desirable to divide the frequency distribution intervals into equal intervals.

[0154] In step S1205, the distance information generation unit 322 uses the most frequently occurring distance information from the frequency distribution as object distance information indicating the distance of the Nth object.

[0155] Alternatively, the average of the distance values ​​contained within the region can be calculated and used as object distance information. When calculating the average, a weighted average can be used using confidence data. By setting a larger weight for each pixel as the confidence level of the distance value increases, the object distance can be calculated with greater accuracy.

[0156] Furthermore, it is desirable that the object distance information be information indicating the distance from a predetermined position on the vehicle 100 to the object, in order to facilitate path generation in the path generation process described later. When using a distance value D as the distance information, since the distance value D indicates the distance from the image sensor 302 to the object, the most frequent value can be offset by a predetermined amount to obtain information indicating the distance from a predetermined position on the vehicle 100 (for example, the front end) to the object. When using a defocus amount ΔL as the distance information, it is converted to the distance from the image sensor 302 using equation 6, and then offset by a predetermined amount to obtain information indicating the distance from a predetermined position on the vehicle 100 to the object.

[0157] In step S1206, the distance information generation unit 322 determines whether N+1 is greater than the number of detected objects Nmax. If N+1 is less than the number of detected objects Nmax (the answer is No in step S1206), then in step S1207, the distance information generation unit 322 sets N to N+1, and the process returns to step S1203. That is, object distance information is extracted for the next object (the N+1th object). If N+1 is greater than the number of detected objects Nmax in step S1206 (the answer is Yes in step S1206), the process proceeds to step S1208.

[0158] In step S1208, the distance information generation unit 322 outputs object distance information for each of the Nmax objects, along with time information, to the path generation unit 330, and terminates processing. This information is stored in memory 180. Distance information from the radar device 120 is also stored in memory 180 along with time information. The flow shown in Figure 12 is repeated periodically until the user issues a termination instruction (not shown).

[0159] The object distance information generation process described above generates object distance information for each object included in the external information. In particular, by statistically determining the object distance information from the distance information contained in the region of the distance image 820 corresponding to the object detected in image 810, it is possible to suppress variations in pixel-by-pixel distance information caused by noise and calculation accuracy. Therefore, it is possible to obtain information indicating the distance of an object with higher accuracy. As mentioned above, the method for statistically determining the distance information is to take the most frequent distance information, mean, median, etc., from the distribution of distance information, and various methods can be employed.

[0160] Next, we will explain the process (route generation process) that generates route information, which is performed by the route generation unit 330 of the route generation ECU 130. Route information includes information such as the direction of travel and speed of the vehicle. Route information can also be said to be operation plan information. The route generation unit 330 outputs the route information to the vehicle control ECU 140. Based on the route information, the vehicle control ECU 140 controls the direction of travel and speed of the vehicle by controlling the drive unit 170.

[0161] In this embodiment, the route generation unit 330 generates route information so that the vehicle 100 follows another vehicle (a vehicle in front) when there is another vehicle (a vehicle in front) in the direction of travel of the vehicle 100. Furthermore, the route generation unit 330 generates route information so that the vehicle 100 takes evasive action to avoid colliding with an object.

[0162] Figure 13 is a flowchart showing an example of route generation processing performed by the route generation unit 330 according to the first embodiment. Note that the computer included in the route generation device 150 executes a computer program stored in memory to perform each step of the flowchart in Figure 13.

[0163] The route generation unit 330, for example, when instructed by the user to start the route generation process, starts the flow shown in Figure 13 and generates route information for the vehicle 100 based on external information, object distance information, and distance information generated by the radar device 120. The route generation ECU 130 reads the external information, object distance information, and distance information generated by the radar device 120 for each time point from the memory 180 of the route generation device 150, for example, and processes it.

[0164] In step S1301, the path generation unit 330 detects objects on the planned travel path of the vehicle 100 from external information and object distance information. The path generation unit 330 determines objects on the travel path by comparing the direction of travel planned by the vehicle 100 with the position and type of objects included in the external information. The direction of travel planned by the vehicle 100 is determined based on information related to the control of the vehicle 100 (steering angle, speed, etc.) obtained from the vehicle control ECU 140. If no objects are detected on the travel path, the path generation unit 330 determines "no objects".

[0165] For example, suppose the imaging device 110 acquires the image 810 shown in Figure 8(A). If the path generation unit 330 determines from the control information of the vehicle 100 acquired from the vehicle control ECU 140 that the vehicle 100 is moving in the direction along lane 805, the path generation unit 330 detects the vehicle 802 as an object on the travel path.

[0166] In steps S1302 and S1303, the path generation unit 330 determines whether to generate path information for follow-the-object driving or path information for avoidance driving, based on the distance between the vehicle 100 and the object on the driving path and the speed Vc of the vehicle 100.

[0167] In step S1302, the path generation unit 330 determines whether the distance between the vehicle 100 and an object on the travel path is shorter than a threshold Dth. The threshold Dth is expressed as a function of the vehicle's travel speed Vc. The higher the travel speed, the larger the threshold Dth. If the path generation unit 330 determines that the distance between the vehicle 100 and an object on the travel path is shorter than a threshold Dth (step S1302 Yes), the process proceeds to step S1303. If the path generation unit 330 determines that the distance between the vehicle 100 and an object on the travel path is greater than or equal to the threshold Dth (step S1302 No), the process proceeds to step S1308.

[0168] In step S1303, the route generation unit 330 determines whether the relative speed between the vehicle 100 and the object on the travel path is a positive value. The route generation unit 330 obtains the identification number of the object on the travel path from external information and obtains object distance information for the object on the travel path at each time from external information obtained from the present time up to a predetermined time ago. The route generation unit 330 calculates the relative speed between the vehicle 100 and the object on the travel path from the object distance information for the object on the travel path during the period up to the predetermined time ago. If the relative speed (vehicle speed 100 - speed of the object on the travel path) is a positive value, it indicates that the vehicle 100 and the object on the travel path are approaching each other.

[0169] If the path generation unit 330 determines that the relative speed between the vehicle 100 and the object on the travel path is a positive value (step S1303 Yes), the process proceeds to step S1304. If the path generation unit 330 determines that the relative speed between the vehicle 100 and the object on the travel path is not a positive value (step S1303 No), the process proceeds to step S1308.

[0170] If the process proceeds to step S1304, route information for executing evasive action is generated. If the process proceeds to step S1308, route information for performing follow-up driving that maintains a distance from the vehicle in front is generated.

[0171] In other words, the path generation unit 330 performs an evasive action when the distance between the vehicle 100 and the object on the travel path is shorter than the threshold Dth, and the relative speed between the vehicle 100 and the object on the travel path is a positive value. On the other hand, the path generation unit 330 determines to follow the object if the distance between the vehicle 100 and the object on the travel path is greater than or equal to the threshold Dth. Alternatively, the path generation unit 330 determines to follow the object if the distance between the vehicle 100 and the object on the travel path is shorter than the threshold Dth, and the relative speed between the vehicle 100 and the object on the travel path is zero or a negative value (moving away).

[0172] If the distance between vehicle 100 and an object on the travel path is shorter than a threshold Dth calculated from the speed of vehicle 100, and the relative speed between vehicle 100 and the object on the travel path is a positive value, then there is a high probability that vehicle 100 will collide with the object on the travel path. Therefore, the path generation unit 330 generates path information to take evasive action. Otherwise, the path generation unit 330 takes follow-the-object driving action. In addition, the above determination may include a determination of whether the detected object on the travel path is a moving object (such as a car or motorcycle).

[0173] In step S1304, the path generation unit 330 starts the process of generating path information for executing an avoidance action.

[0174] In step S1305, the path generation unit 330 acquires information regarding avoidance spaces. The path generation unit 330 acquires distance information from the radar device 120, including the distance to objects to the sides and rear of the vehicle 100, and predicted distance information to those objects.

[0175] The path generation unit 330 acquires information indicating the direction and size of the space around the vehicle 100 in which the vehicle 100 can move, based on distance information (fourth distance information) acquired from the radar device 120, the speed of the vehicle 100, and information indicating the size of the vehicle 100. In this embodiment, distance information (fourth distance information) acquired from the radar device 120 is used for avoidance space, but it may also be used to generate integrated distance information to an object.

[0176] In step S1306, the path generation unit 330 sets path information for avoidance action based on information indicating the direction and size of the space in which the vehicle 100 can move, external information, and object distance information. The path information for avoidance action is, for example, information to change the vehicle 100's course to the right, for example, while decelerating, if there is a space to avoid to the right of the vehicle 100.

[0177] In step S1307, the route generation unit 330 outputs route information to the vehicle control ECU 140. The vehicle control ECU 140 determines parameters to control the drive unit 170 based on the route information so that the vehicle 100 travels along the route indicated in the acquired route information, and controls the drive unit 170. Specifically, the vehicle control ECU 140 determines the steering angle, accelerator control value, brake control value, control signal for gear engagement, and lamp illumination control signal based on the route information.

[0178] Furthermore, step S1307 functions as a route generation step (route generation means) that generates route information based on distance information. Meanwhile, in step S1308, the route generation unit 330 starts the process of generating route information for executing follow operation.

[0179] In step S1309, the route generation unit 330 generates route information so that the vehicle 100 follows an object (a preceding vehicle) on its travel path. Specifically, the route generation unit 330 generates route information so that the distance between the vehicle 100 and the preceding vehicle (inter-vehicle distance) remains within a predetermined range. For example, if the relative speed between the vehicle 100 and the preceding vehicle is zero or a negative value, or if the inter-vehicle distance is greater than or equal to a predetermined distance, the route generation unit 330 generates route information so that the vehicle 100 maintains a straight direction of travel while accelerating and decelerating to maintain the predetermined inter-vehicle distance.

[0180] Furthermore, the route generation unit 330 generates route information so that the vehicle 100's driving speed does not exceed a predetermined value (for example, the legal speed limit on the road the vehicle 100 is traveling on, or a set driving speed based on instructions from the driver 101). After step S1309, the process proceeds to step S1307, where the vehicle control ECU 140 controls the drive unit 170 based on the generated route information. Here, the vehicle control ECU 140 functions as a driving control means that controls the driving state based on the route information generated by the route generation unit 330, based on information such as the position and size of objects detected by the object detection unit 323.

[0181] Next, in step S1310, it is determined whether the user has issued an instruction to terminate the route generation process. If yes, the route generation unit 330 terminates the route information generation process. If no, the process returns to step S1301 and the route information generation process is repeated.

[0182] According to the control described above, by statistically processing distance information within a frame based on the position and size of objects in the image, the effects of sensor noise and local distance errors caused by high-luminance reflections from the subject can be reduced, and the distance value to each object can be calculated with high accuracy. Furthermore, since the distance value to each object can be calculated with high accuracy, the route generation ECU 130 can calculate the route of vehicle 100 with high accuracy, enabling vehicle 100 to drive more stably.

[0183] <Example 1> In the process described above, the object's region was shown as a rectangular area (object frame) encompassing the object. However, the object's region may also be defined as an area with the shape of the object, bounded by the outer perimeter of the object within the image. In that case, in step S702, the object detection unit 323 stores the area in the image 810 where the object exists as the object's region in the external information. For example, the object detection unit 323 can divide the image into regions for each object by identifying attributes for each pixel of the image data.

[0184] Figure 8(E) is a schematic diagram showing an example in which the object detection unit 323 divides the image into regions for each object and superimposes the results onto the image 810. Region 831 represents the region of person 801, region 832 represents the region of vehicle 802, and region 833 represents the region of sign 803. Furthermore, region 834 represents the region of road 804, and region 835 represents the region of lane 805.

[0185] In this case, in steps S1203 and S1204, the distance information generation unit 322 can calculate, for example, the frequency distribution of distance values ​​contained within each region shown in Figure 8(E) for each object.

[0186] By defining the object's domain in this way, distance information from non-object elements such as the background becomes less likely to be included within the domain. In other words, the distribution of distance information within the domain becomes more accurately reflected in the object's distance information. Therefore, the influence of non-object elements such as the background and foreground can be reduced, allowing for more accurate calculation of the object's distance value.

[0187] <Modification 2> In this embodiment, the imaging device 110 sequentially outputs image information and distance image information. Furthermore, the object detection unit 323 sequentially generates external information using the sequentially received image information. The external information includes the identification number of the object, and if an object with the same identification number is detected at a certain time T0 and time T1, it is possible to determine the time changes of the distance information and detected size of the object. Accordingly, in the modified example 2, the distance information generation unit 322 calculates the average of the distance values ​​D of objects with the same identification number within a predetermined time range. This reduces distance variation in the time direction.

[0188] Figure 14 is a flowchart showing an example of object distance information generation processing performed by the distance information generation unit 322 in modified example 2. The computer included in the path generation device 150 executes a computer program stored in memory to perform each step of the flowchart in Figure 14. For each process in this flowchart, those indicated with the same number as those shown in Figure 12 are the same as those described in Figure 12, and therefore their explanation is omitted.

[0189] In step S1401, the distance information generation unit 322 selects the most frequently occurring distance information from the frequency distribution generated in step S1204 and uses it as object distance information indicating the distance to the Nth object. The distance information generation unit 322 then stores (remembers) the object distance information in the memory 340 along with the identification number and time.

[0190] In step S1402, the distance information generation unit 322 retrieves the history of object distance information with the same identification number as the Nth object from the object distance information stored in the memory 340. The distance information generation unit 322 retrieves object distance information with the same identification number corresponding to a predetermined time before the time corresponding to the latest object distance information. Figure 15 is a schematic diagram illustrating an example of the time change of object distance information for an object with the same identification number as the Nth object. The horizontal axis shows time, and the vertical axis shows object distance information (distance value D). Time t0 indicates the time when the latest distance value D was acquired.

[0191] In step S1403, the distance information generation unit 322 calculates the average value of object distance information included in a time range from the time the latest object distance information was acquired to a predetermined time before, based on the history of object distance information for objects with the same identification number as the acquired Nth object. For example, in Figure 15, the distance information generation unit 322 calculates the average value of the distance values ​​of four points included in the predetermined time range ΔT.

[0192] As described above, by using the identification number included in the external information to obtain a history of object distance information (distance value) for the same object and taking a time average, it is possible to suppress variability. Even if the road on which vehicle 100 travels changes (for example, curves, slopes, rough roads with many bumps, etc.), it is possible to track the same object and calculate the average value in the time direction.

[0193] Therefore, while reducing the effects of changes in the driving environment, it is possible to reduce the variation in distance values ​​due to noise such as optical shot noise contained in the image signal and calculate the distance value of an object with greater accuracy. In the modified example 2, in order to obtain a similar effect, when acquiring object distance information, it is also possible to acquire object distance information that has been averaged over time to some extent by passing it through a low-pass filter.

[0194] <Other Embodiments> In the embodiments described above, an imaging device was explained as an example in which left and right parallax images are acquired using a phase-difference imaging method via the same optical system to acquire distance image data. However, the method of acquiring parallax images is not limited to this. It is also possible to acquire left and right parallax images using a so-called stereo camera, in which a left parallax image and a right parallax image are acquired respectively by two imaging devices placed at a predetermined distance apart.

[0195] Furthermore, it is also possible to perform the aforementioned distance measurement using distance measurement devices such as LiDAR to acquire distance information and external information obtained through image recognition of the captured images obtained from the imaging device.

[0196] Furthermore, in the above embodiments, integrated distance information may be generated based on at least two histories of the first to third distance information of an object or the history of integrated distance information.

[0197] Furthermore, the above embodiments may be combined as appropriate, including, for example, the following configurations.

[0198] (Configuration 1) An image processing apparatus characterized by comprising: boundary detection means for dividing image information into a plurality of divided regions extending in a predetermined direction and detecting boundaries based on at least color information for the image within each divided region; and integrated distance equivalent information determination means for determining integrated distance equivalent information for each sub-region separated by the boundary within the divided region based on distance equivalent information for each pixel of the image information.

[0199] (Configuration 2) An image processing apparatus characterized by having a classification means that performs a predetermined classification of the sub-regions, for which the integrated distance equivalent information has been determined, according to the integrated distance equivalent information of the sub-regions.

[0200] (Composition 3) The image processing apparatus according to configuration 2, characterized in that the classification means classifies the sub-region into one of three categories: sky, road, or other.

[0201] (Composition 4) The image processing apparatus according to any one of configurations 1 to 3, characterized in that the predetermined direction is the vertical direction of the screen.

[0202] (Composition 5) An image processing apparatus according to any one of configurations 1 to 4, characterized by having distance equivalent information measuring means for detecting the distance equivalent information for each pixel of the image information.

[0203] (Composition 6) The image processing apparatus according to configuration 5, characterized in that the distance equivalent information measuring means generates a confidence score representing the certainty of the distance equivalent information of each pixel based on the color information, and the integrated distance equivalent information determining means determines the integrated distance equivalent information for each sub-region based on the distance equivalent information of each pixel weighted based on the confidence score.

[0204] (Composition 7) The image processing apparatus according to configuration 6, comprising an imaging means for capturing an image of a subject and generating multiple images with different viewpoints, wherein the distance equivalent information measuring means calculates the amount of parallax of the multiple images based on the color information, calculates a parallax confidence level indicating the certainty of the amount of parallax of each pixel based on the color information, and generates a confidence level based on the parallax confidence level.

[0205] (Composition 8) The image processing apparatus according to configuration 7, characterized in that the distance equivalent information measuring means calculates the parallax reliability of pixels in the plurality of images whose brightness value is greater than or equal to a predetermined value, such that it is lower than when the brightness value is less than the predetermined value.

[0206] (Composition 9) The image processing apparatus according to configuration 5, comprising an imaging means for capturing an image of a subject and generating the image information, wherein the distance equivalent information measuring means measures the distance equivalent information using a phase difference distance measurement method based on signals from a first photoelectric conversion unit and a second photoelectric conversion unit arranged within the pixels of the imaging means.

[0207] (Composition 10) The image processing apparatus according to configuration 5, characterized in that the distance equivalent information measuring means measures the distance equivalent information using a phase difference distance measurement method based on two image signals from a stereo camera.

[0208] (Composition 11) The image processing apparatus according to any one of configurations 1 to 10, characterized in that the boundary detection means detects the position where the color information changes from a predetermined threshold as the boundary.

[0209] (Composition 12) The image processing apparatus according to configuration 11, characterized in that the divided region has a plurality of pixels in a direction intersecting the predetermined direction, and the color information is a representative value obtained based on the plurality of pixels.

[0210] (Composition 13) The image processing apparatus according to either configuration 2 or 3, characterized in that it has an object detection means that integrates images of a plurality of divided regions, including the sub-regions classified by the classification means, to perform object detection.

[0211] (Composition 14) A mobile body comprising: boundary detection means for dividing image information into a plurality of divided regions extending in a predetermined direction and detecting boundaries based on at least color information for the image within each divided region; integrated distance equivalent information determination means for determining integrated distance equivalent information for each sub-region separated by the boundary within the divided region based on distance equivalent information for each pixel of the image information; object detection means for performing object detection by integrating images of a plurality of divided regions including the sub-regions for which the integrated distance equivalent information has been determined; and driving control means for controlling the driving state based on information of an object detected by the object detection means.

[0212] (Method 1) An image processing method characterized by comprising: a boundary detection step of dividing image information into a plurality of divided regions extending in a predetermined direction and detecting a boundary based on at least color information for the image within each divided region; and an integrated distance equivalent information determination step of determining integrated distance equivalent information for each sub-region separated by the boundary within the divided region, based on distance equivalent information for each pixel of the image information.

[0213] (Program 1) A computer program for controlling, by computer, each means of an image processing apparatus described in any one of configurations 1 to 9 or a mobile body described in configuration 10.

[0214] In the above-described embodiment, an example was given in which a distance calculation device as an electronic device was mounted on a vehicle, which was the mobile object. However, the mobile object can be anything that is mobile, such as a motorcycle, bicycle, wheelchair, ship, airplane, drone, or mobile robot such as an AGV or AMR. Furthermore, the distance calculation device as an electronic device in this embodiment is not limited to those mounted on such mobile objects, but also includes devices that acquire images from a camera or other device mounted on the mobile object via communication and calculate the distance at a location away from the mobile object.

[0215] Although the present invention has been described in detail above based on preferred embodiments (examples), the present invention is not limited to the above embodiments (examples), and various modifications are possible in accordance with the spirit of the present invention, and these modifications are not excluded from the scope of the present invention.

[0216] Furthermore, the present invention may also be realized by supplying a storage medium containing software program code (control program) that realizes the functions of the embodiments described above to a system or device. It can also be realized by the computer (or CPU or MPU) of the system or device reading and executing the computer-readable program code stored on the storage medium.

[0217] In that case, the program code read from the storage medium itself will realize the function of the embodiment described above, and the storage medium storing that program code will constitute the present invention. [Explanation of symbols]

[0218] 100: Vehicles 101: Driver 110: Imaging device 120: Radar equipment 130: Route generation ECU 140: Vehicle control ECU 150: Route Generator 160: Vehicle Information Measuring Instrument 320: Object information generation unit 321: Boundary Processing Unit 322: Distance information generation unit 323: Object detection unit

Claims

1. A region division means for dividing image information into multiple divided regions extending in a predetermined direction, Boundary detection means for detecting boundaries based on color information for images within each of the divided regions, Integrated distance equivalent information determination means that, based on the distance equivalent information of each pixel of the image information, integrates the distance equivalent information for each sub-region separated by the boundary within the divided region to determine integrated distance equivalent information indicating the respective distance of each sub-region, A classification means that performs a predetermined classification of the sub-regions in accordance with the integrated distance equivalent information of the sub-regions, An image processing apparatus characterized by having

2. The image processing apparatus according to claim 1, characterized in that the classification means classifies the sub-region into one of three categories: sky, road, and other.

3. The image processing apparatus according to claim 1, characterized in that the boundary detection means divides the image information into a plurality of divided regions that extend vertically from the pixels at the top edge of the screen to the pixels at the bottom edge.

4. The image processing apparatus according to claim 1, characterized in that it has distance equivalent information measuring means for detecting distance equivalent information for each pixel of the image information.

5. The distance equivalent information measuring means generates a confidence level representing the certainty of the distance equivalent information of each pixel based on the color information, The image processing apparatus according to claim 4, wherein the integrated distance equivalent information determination means determines the integrated distance equivalent information for each sub-region based on the distance equivalent information of each pixel weighted based on the confidence level.

6. It has an imaging means that captures an image of a subject and generates multiple images from different viewpoints, The image processing apparatus according to claim 5, wherein the distance equivalent information measuring means calculates the amount of parallax of the plurality of images based on the color information, calculates a parallax confidence level indicating the certainty of the amount of parallax of each pixel based on the color information, and generates the confidence level based on the parallax confidence level.

7. The image processing apparatus according to claim 6, characterized in that the distance equivalent information measuring means calculates the parallax reliability of pixels in the plurality of images whose brightness value is greater than or equal to a predetermined value, such that it is lower than when the brightness value is less than the predetermined value.

8. The system includes an imaging means that captures an image of a subject and generates the image information, The image processing apparatus according to claim 4, characterized in that the distance equivalent information measuring means measures the distance equivalent information by a phase difference distance measurement method based on signals from a first photoelectric conversion unit and a second photoelectric conversion unit arranged in the pixels of the imaging means.

9. The image processing apparatus according to claim 4, characterized in that the distance equivalent information measuring means measures the distance equivalent information by a phase difference distance measuring method based on two image signals from a stereo camera.

10. The image processing apparatus according to claim 1, characterized in that the boundary detection means detects the position where the color information changes from a predetermined threshold as the boundary.

11. The divided region has a plurality of pixels in a predetermined direction, The image processing apparatus according to claim 10, characterized in that the color information is a representative value obtained based on the plurality of pixels.

12. The image processing apparatus according to claim 1, further comprising an object detection means that performs object detection by integrating images of a plurality of divided regions, including the sub-regions that have been classified by the classification means.

13. A region division means for dividing image information into multiple divided regions extending in a predetermined direction, Boundary detection means for detecting boundaries based on color information for images within each of the divided regions, Integrated distance equivalent information determination means that, based on the distance equivalent information of each pixel of the image information, integrates the distance equivalent information for each sub-region separated by the boundary within the divided region to determine integrated distance equivalent information indicating the respective distance of each sub-region, A classification means that performs a predetermined classification of the sub-regions in accordance with the integrated distance equivalent information of the sub-regions, A mobile body characterized by having the following features.

14. A region segmentation step in which image information is divided into multiple segmented regions extending in a predetermined direction, A boundary detection step that detects at least a boundary based on color information for the image within each of the divided regions, A step to determine integrated distance equivalent information, in which the distance equivalent information of each pixel of the image information is integrated for each sub-region separated by the boundary within the divided region, and integrated distance equivalent information indicating the distance of each sub-region is determined. A classification step in which the sub-regions for which the integrated distance equivalent information has been determined are classified according to the integrated distance equivalent information of the sub-regions, An image processing method characterized by having the following features.

15. A computer program for controlling, by computer, each means of an image processing apparatus according to any one of claims 1 to 12 or a moving body according to claim 13.

Images