Electronic devices, mobile devices, imaging devices, and control methods, programs, and storage media for electronic devices.

Abstract

To provide an electronic apparatus, a movable body, an imaging apparatus, and a control method for an electronic apparatus, and a program that can accurately acquire the distance to an object included in an image.SOLUTION: A route generation device 150 according to the present invention has: a recognition unit 321 that sets an object area corresponding to an object included in an image acquired by an image pickup device 302; a distance image creation unit 312 that acquires a distance map corresponding to the image and having distance information on pixels; and a distance information generation unit 322 that, by using the distance information on pixels corresponding to the object area in the distance map, acquires object distance information indicating the distance to the object.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an electronic device that acquires the distance from a moving object to an object, a moving object, an imaging device, a control method for an electronic device, a program, and a storage medium.

Background Art

[0002] There is a photographing apparatus having a sensor including a pixel region in which pixels having a photoelectric conversion function are two-dimensionally arranged, and capable of acquiring an image signal and distance information in each pixel region. The solid-state imaging device described in Patent Document 1 arranges pixels having a distance measuring function in some or all of the pixels of the imaging device, and detects the subject distance based on the phase difference detected on the imaging surface (imaging surface phase difference method). A positional shift is calculated based on the correlation between two image signals based on images generated by light fluxes passing through different pupil regions of the imaging optical system included in the photographing apparatus, and the distance is acquired based on the positional shift.

[0003] The correlation between the two image signals is evaluated using a method such as a region-based matching method in which image signals included in a predetermined collation region are cut out from each image signal and the correlation is evaluated.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When the contrast change of the subject included in the image signal is small, or when the amount of noise included in the image signal is large, etc., misevaluation of the correlation may occur due to the subject or shooting conditions. If the occurrence of misevaluation of the correlation is included more than a certain level, the amount of positional shift between the two calculated image signals has an error, and the accuracy of the acquired distance may decrease.

[0006] The present invention aims to provide electronic equipment, a mobile device, an imaging device, and a control method and program for electronic equipment that enable the acquisition of the distance to an object contained in an image with high precision. [Means for solving the problem]

[0007] One embodiment of the electronic device according to the present invention is characterized by comprising: setting means for setting an object region corresponding to an object included in an image acquired from an imaging means; acquisition means for acquiring a distance map corresponding to the image and having distance information for each pixel; and processing means for acquiring object distance information indicating the distance to the object using the distance information of the pixels corresponding to the object region in the distance map. [Effects of the Invention]

[0008] According to the present invention, the electronic device, mobile device, imaging device, and control method and program for the electronic device make it possible to acquire the distance to an object included in an image with high precision. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram showing the configuration of the vehicle. [Figure 2] This is a diagram showing the configuration of the vehicle. [Figure 3] This is a block diagram showing the configuration of a route generation device. [Figure 4] This is a schematic diagram showing the configuration of an image sensor. [Figure 5] This is a schematic diagram showing the relationship between subject distance and incident light in the image plane phase-detection imaging system. [Figure 6] This flowchart shows the processes performed by the image processing unit. [Figure 7] This is a flowchart showing the processes performed by the distance information generation unit. [Figure 8] This is a schematic diagram showing the images and information used in the processing performed by the distance information generation unit. [Figure 9]It is a flowchart showing the route generation process executed by the route generation unit. [Figure 10] It is a flowchart showing the object distance information generation process performed by the distance information generation unit. [Figure 11] It is a schematic diagram showing the temporal change of the object distance information of an object with the same identification number as the Nth object. [Figure 12] It is a flowchart showing the object distance information generation process performed by the distance information generation unit. [Figure 13] It is a schematic diagram for explaining weighted average. [Figure 14] It is a flowchart showing the process for acquiring the distance measurement value (object distance information). [Figure 15] It is a flowchart showing the process when a plurality of input data continuous in time series are input. [Figure 16] It is a schematic diagram for explaining the correction process of the absolute error. [Figure 17] It is a block diagram showing a configuration example of the distance measurement system. [Figure 18] It is a schematic diagram showing the output results of the section line detection task and the object recognition task executed by the object detection unit. [Figure 19] It is a diagram for explaining the positional relationship between the camera mounting position and the road surface. [Figure 20] It is a schematic diagram showing a scene in which two signs with known object sizes are detected on the road. [Figure 21] It is a block diagram showing a configuration example of the scaling distance measurement unit. [Figure 22] It is a schematic diagram schematically showing the processes of lane width detection and lane center detection executed by the lane analysis unit. [Figure 23] It is a schematic diagram comparing the positions indicating the lane width in the captured image with respect to the presence or absence of the roll angle. [Figure 24] It is a flowchart showing the roll angle estimation process executed by the roll angle estimation unit. [Figure 25] It is a schematic diagram for explaining each process in the roll angle estimation process. [Figure 26]This flowchart shows the coordinate estimation process for lane width data performed by the ground contact position estimation unit. [Figure 27] This is a schematic diagram showing the coordinates of the grounding position of the distance measurement target, as set by the grounding position estimation unit. [Figure 28] This flowchart shows the distance estimation process to the object to be measured, which is performed by the object distance calculation unit. [Figure 29] This is a schematic diagram showing the lane width data for each lane. [Modes for carrying out the invention]

[0010] <First Embodiment> The first embodiment of the present invention will be described in detail below with reference to the figures.

[0011] In the following description, a route generation device (electronic device) equipped with a camera will be used as an example of the route generation device of the present invention, but the application of the present invention is not limited to this.

[0012] Furthermore, in explanations that refer to diagrams, the same reference numeral will be used for parts that represent the same component, even if the diagram numbers differ, and redundant explanations will be avoided as much as possible.

[0013] Figure 1 is a schematic diagram showing the configuration of vehicle 100. Vehicle 100 is a mobile unit comprising an imaging device 110, a radar device 120, a route generation ECU 130, a vehicle control ECU 140, and a group of measuring instruments 160. Vehicle 100 also comprises a drive unit 170, a memory 180, and a memory 190. The drive unit 170, memory 180, and memory 190 will be explained using Figure 2. The imaging device 110, radar device 120, route generation ECU 130, vehicle control ECU 140, and group of measuring instruments 160 are included. The imaging device 110 and the route generation ECU 130 constitute a route generation device 150. Vehicle 100 is capable of carrying a driver 101, and when in motion, the driver 101 is seated facing forward (direction of travel) of vehicle 100. The driver 101 can control the vehicle's operation by operating the steering wheel, accelerator pedal, brake pedal, and other control components of the vehicle 100.

[0014] The imaging device 110 is positioned to photograph the area in front of the vehicle 100 (the normal direction of travel). As shown in Figure 1, the imaging device 110 is positioned near the top edge of the windshield of the vehicle 100 and photographs an area within a predetermined angular range (hereinafter referred to as the shooting angle) toward the front of the vehicle 100. The imaging device 110 may also be positioned to photograph the area behind the vehicle 100 (opposite to the normal direction of travel, in the reverse direction) or to photograph the side. Multiple imaging devices 110 may be positioned on the vehicle 100.

[0015] Figure 2 is a diagram showing the configuration of vehicle 100.

[0016] The imaging device 110 captures the surrounding environment of the vehicle (surrounding environment), including the road on which the vehicle 100 travels (travel road). The imaging device 110 detects objects that are within the field of view of the imaging device 110. The imaging device 110 acquires information about the detected object (external information) and information about the distance to the detected object (object distance information), and outputs it to the route generation ECU 130. The object distance information can be any information that can be converted to the distance from a predetermined position on the vehicle 100 to the object using a predetermined reference table or predetermined conversion coefficients and conversion formulas. For example, the distance may be assigned to a predetermined integer value and output sequentially to the route generation ECU 130.

[0017] The imaging device 110 has a sensor equipped with a pixel region in which pixels having a photoelectric conversion function are arranged in two dimensions, and is capable of acquiring the distance of an object using the image plane phase difference method. The acquisition of object distance information using the image plane phase difference method will be described later.

[0018] The radar device 120 is a detection device that detects objects by transmitting electromagnetic waves as transmission waves and receiving the reflected waves. Based on the time from the transmission of electromagnetic waves to the reception of the reflected waves and the received intensity of the reflected waves, the radar device 120 acquires distance information indicating the distance to the object in the direction of electromagnetic wave transmission. The radar device 120 outputs the distance information to the path generation ECU 130. The radar device 120 is assumed to be a millimeter-wave radar device that uses electromagnetic waves with wavelengths ranging from the millimeter-wave band to the submillimeter-wave band.

[0019] In this embodiment, the vehicle 100 is equipped with multiple radar devices 120, for example, radar devices 120 are mounted on the left and right sides of the front of the vehicle 100, and on the left and right sides of the rear of the vehicle 100. Each radar device 120 scans within a predetermined angular range with electromagnetic waves, and based on the time from the transmission of electromagnetic waves to the reception of reflected waves and the received intensity of the reflected waves, the distance from the radar device 120 is measured and distance information at the scanning position is generated. In addition to the distance from the radar device 120, the distance information may include information on the received intensity of the reflected waves and the relative velocity of the object.

[0020] The measuring instrument group 160 includes a speed measuring instrument 161, a steering angle measuring instrument 162, and an angular velocity measuring instrument 163, and acquires vehicle information related to the vehicle's state, such as travel speed, steering angle, and angular velocity. The speed measuring instrument 161 is a measuring instrument that detects the travel speed of the vehicle 100. The steering angle measuring instrument 162 is a measuring instrument that detects the steering angle of the vehicle 100. The angular velocity measuring instrument 163 is a measuring instrument that detects the angular velocity of the vehicle 100 in the turning direction. Each measuring instrument outputs a measurement signal corresponding to the measured parameter as vehicle information to the route generation ECU 130.

[0021] The route generation ECU 130 is configured using logic circuits and generates the vehicle 100's travel trajectory and route information related to the travel trajectory based on measurement signals, external information, object distance information, and distance information. The route generation ECU 130 outputs to the vehicle control ECU 140. The data processed and programs executed by the route generation ECU 130 are stored in memory 180.

[0022] Here, the travel trajectory is information indicating the path (route) that vehicle 100 will take. The route information is information that allows vehicle 100 to travel along the route indicated by the travel trajectory.

[0023] The vehicle control ECU 140 is configured using logic circuits and controls the drive unit 170 so that the vehicle 100 travels along the route specified in the route information, based on route information and vehicle information acquired from the measuring instrument group 160. The data processed and the programs executed by the route generation ECU 130 are stored in the memory 190.

[0024] The drive unit 170 is a drive component for driving the vehicle and includes a power unit (not shown), such as an engine or motor, which generates energy to rotate the tires, and a steering unit that controls the direction of travel of the vehicle. The drive unit 170 also includes a gearbox for rotating the tires using the energy generated by the power unit, a gear control unit that controls the configuration within the gearbox, and a brake unit that performs braking. The vehicle control ECU 140 controls the drive unit 170 so that the vehicle travels along a path corresponding to the path information, and adjusts the amount of drive, braking, steering, etc. of the vehicle 100. Specifically, the vehicle control ECU 140 operates the vehicle 100 by controlling the brakes, steering, gear configuration, etc.

[0025] Furthermore, the route generation ECU 130 and the vehicle control ECU 140 may be configured with a common central processing unit (CPU) and memory for storing calculation processing programs.

[0026] The HMI240 is a Human Machine Interface for conveying information to the driver 101. The HMI240 includes a display that is visible when the driver 101 is in the driving position, and a display control device that generates information to be displayed on the display. The HMI240 also includes a device that outputs sound (speaker system) and a voice control device that generates voice data. The display control device displays navigation information on the display based on the route information generated by the route generation ECU 130. The voice control device generates voice data to notify the driver 101 of information based on the route information and outputs it from the speaker system. The voice data may include, for example, data to notify the driver that an intersection to turn is approaching.

[0027] Figure 3 is a block diagram showing the configuration of the route generation device 150. The route generation device 150 comprises an imaging device 110 and a route generation ECU 130.

[0028] The imaging device 110 includes an imaging optical system 301, an image sensor 302, an image processing unit 310, and an object information generation unit 320. The imaging optical system 301, image sensor 302, image processing unit 310, and object information generation unit 320 are arranged inside the housing (not shown) of the imaging device 110.

[0029] The imaging optical system 301 is the imaging lens of the imaging device 110 and has the function of forming an image (optical image) of the subject on the image sensor 302. The imaging optical system 301 is composed of multiple lens groups. The imaging optical system 301 has an exit pupil at a predetermined distance from the image sensor 302.

[0030] The image sensor 302 is composed of CMOS (complementary metal-oxide-semiconductor) or CCD (charge-coupled device) and has a distance measurement function using an image plane phase-difference distance measurement method. The image sensor 302 sequentially outputs an image signal based on the image formed by the imaging optical system 301 to the image processing unit 310. The image sensor 302 has a pixel region in which pixels having a photoelectric conversion function are arranged in two dimensions. Each pixel region has two photoelectric conversion units (a first photoelectric conversion unit and a second photoelectric conversion unit). The image sensor 302 photoelectrically converts the subject image formed on the image sensor 302 via the imaging optical system 301 to generate an image signal based on the subject image. The image sensor 302 outputs the image signal to the image processing unit 310. The image signal is a signal consisting of the output values ​​of each photoelectric conversion unit in each pixel region. The image sensor 302 outputs a first image signal based on the signal output from the first photoelectric conversion unit and a second image signal based on the signal output from the second photoelectric conversion unit to the image processing unit 310.

[0031] The image processing unit 310 generates image data containing brightness information for each pixel (red, green, and blue) and distance image data indicating distance information for each pixel, based on the image signal. The image processing unit 310 includes a development unit 311 that generates image data based on the image signal and a distance image generation unit 312 that generates distance image data based on the image signal. The processes performed by these units will be described later. The image processing unit 310 outputs the image data and distance image data to the object information generation unit 320.

[0032] The object information generation unit 320 has a recognition processing unit 321 that detects objects contained in the acquired image based on the image data and generates external information indicating information about the object. The external information includes information indicating the position of the detected object within the image, its size such as width and height, and its region. The external information also includes information about the attributes and identification number of the detected object.

[0033] The object information generation unit 320 includes a distance information generation unit 322 that generates object distance information indicating the distance to an object included in the acquired image, based on external information and distance image data. The object distance information includes information regarding the identification number of the object detected by the external information.

[0034] The object information generation unit 320 outputs external information and object distance information to the path generation ECU 130.

[0035] The image processing unit 310 and the object information generation unit 320 may be composed of one or more processors in the imaging device 110. The functions of the image processing unit 310 and the object information generation unit 320 may be realized by one or more processors executing a program read from the memory 340.

[0036] The route generation ECU 130 includes a route generation unit 330. The route generation unit 330 generates route information based on external information, object distance information, and distance information acquired from the radar device 120.

[0037] Next, the structure and control of each block of the route generation device 150 will be described in detail.

[0038] Figure 4 is a schematic diagram showing the configuration of the image sensor 302. Figure 4(A) is a top view of the image sensor 302 as seen from the direction of light incidence. The image sensor 302 is composed of multiple 2x2 pixel groups 410 arranged in a matrix. Each pixel group 410 has green pixels G1 and G2 for detecting green light, red pixels R for detecting red light, and blue pixels B for detecting blue light. In the pixel group 410, green pixels G1 and G2 are arranged diagonally. Each pixel also has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412.

[0039] Figure 4(B) is a cross-sectional view of the pixel group 410 in the I-I' section of Figure 4(A). Each pixel consists of a microlens 413, a light-receiving layer 414, and a light-guiding layer 415.

[0040] The light guide layer 414 is a light guide member having a microlens 413 for efficiently guiding the light beam incident on the pixel to the light receiving layer 415, a color filter that allows light in a wavelength band corresponding to the color of light detected by each pixel to pass through, and wiring for image readout and pixel driving.

[0041] The light-receiving layer 415 is a photoelectric conversion unit that converts light incident through the light-guide layer 414 into electrical signals and outputs them as electrical signals. The light-receiving layer 415 has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412.

[0042] Figure 5 is a schematic diagram showing the relationship between subject distance and incident light in the image plane phase-difference imaging method. Figure 5(A) is a schematic diagram showing the exit pupil 501 of the imaging optical system 301, the green pixel G1 of the image sensor 302, and the light incident on each light source conversion section of the green pixel G1. The image sensor 302 has multiple pixels, but for simplicity, we will describe one green pixel G1.

[0043] The microlens 413 of the green pixel G1 is arranged such that it is optically conjugate to the exit pupil 501 and the light-receiving layer 415. As a result, the light beam that passes through the first pupil region 510, which is a partial pupil region contained within the exit pupil 501, is incident on the first photoelectric conversion unit 411. Similarly, the light beam that passes through the second pupil region 520, which is a partial pupil region, is incident on the second photoelectric conversion unit 412.

[0044] Each pixel's first photoelectric conversion unit 411 converts the received light beam into electricity and outputs a signal. A first image signal is generated from the signals output from the multiple first photoelectric conversion units 411 contained in the image sensor 302. The first image signal shows the intensity distribution of the image formed on the image sensor 302 by the light beam that mainly passed through the first pupil region 510.

[0045] The second photoelectric conversion unit 412 of each pixel converts the received light beam into electricity and outputs a signal. A second image signal is generated from the signals output from the multiple second photoelectric conversion units 412 included in the image sensor 302. The second image signal shows the intensity distribution of the image formed on the image sensor 302 by the light beam that mainly passed through the second pupil region 520.

[0046] The relative positional shift between the first and second image signals (hereinafter referred to as the disparity amount) is a quantity corresponding to the defocus amount. The relationship between the disparity amount and the defocus amount is explained using Figures 5(B), (C), and (D).

[0047] Figures 5(B), (C), and (D) are schematic diagrams showing the image sensor 302 and the imaging optical system 301. In the figures, 511 represents the first light beam passing through the first pupil region 510, and 521 represents the light beam passing through the second pupil region 520.

[0048] Figure 5(B) shows the state when the image is in focus, with the first light beam 511 and the second light beam 521 converging on the image sensor 302. At this time, the amount of disparity between the first image signal formed by the first light beam 511 and the second image signal formed by the second light beam 521 is 0.

[0049] Figure 5(C) shows a state where the image is defocused in the negative z-axis direction. At this time, the disparity between the first image signal formed by the first light beam 511 and the second image signal formed by the second light beam 521 is not zero, but has a negative value.

[0050] Figure 5(D) shows the state where the image is defocused in the positive z-axis direction. At this time, the disparity between the first image signal formed by the first light beam 511 and the second image signal formed by the second light beam 521 is not zero, but has a positive value.

[0051] A comparison of Figures 5(C) and (D) shows that the direction in which parallax occurs changes depending on whether the amount of defocus is positive or negative. Furthermore, the geometric relationship shows that the amount of parallax corresponds to the amount of defocus. Therefore, as will be described later, the amount of parallax between the first image signal and the second image signal can be detected using a region-based matching method, and the amount of parallax can be converted into the amount of defocus via a predetermined conversion coefficient. In addition, by using the imaging formula of the imaging optical system 301, the amount of defocus on the image side can be converted into the distance to the object.

[0052] The image sensor 302 may output a combined signal of the first image signal and the second image signal, along with the first image signal, to the image processing unit 310. In this case, the image processing unit 310 generates a second image signal based on the difference between the combined signal and the first image signal, and acquires the first image signal and the second image signal.

[0053] Next, the processing performed by the image processing unit 310 will be described.

[0054] Figure 6 is a flowchart showing the processes performed by the image processing unit 310. Figure 6(A) is a flowchart showing the operation of the development process in which the development unit 311 of the image processing unit 310 generates image data from an image signal. The development process is performed in response to the reception of an image signal from the image sensor 302.

[0055] In S601, the developing unit 311 performs a process to generate a composite image signal by combining the first image signal and the second image signal input from the image sensor 302. By combining the first image signal and the second image signal, an image signal based on the image formed by the light beam that has passed through the entire area of ​​the exit pupil 501 can be obtained. When the horizontal pixel coordinate of the image sensor 302 is x and the vertical pixel coordinate is y, the composite image signal Im(x,y) of pixel (x,y) can be expressed by the following equation (Equation 1) using the first image signal Im1(x,y) and the second image signal Im2(x,y).

[0056]

number

[0057] In S602, the development unit 311 performs a correction process for defective pixels in the composite image signal. A defective pixel is a pixel in the image sensor 302 that cannot produce a normal signal output. The development unit 311 acquires information indicating the coordinates of the defective pixels in the image sensor 302. The development unit 311 generates the composite image signal of the defective pixel using a median filter that replaces the defective pixel's composite image signal with the median value of the composite image signals of the pixels surrounding the defective pixel. Alternatively, as a method for correcting the composite image signal of the defective pixel, the signal value of the defective pixel may be generated by interpolating using pre-prepared coordinate information of the defective pixel and the signal values ​​of the pixels surrounding the defective pixel.

[0058] In step S603, the development unit 311 applies light intensity correction processing to the composite image signal to compensate for the reduction in light intensity at the edges of the field of view caused by the imaging optical system 301. As a method of light intensity correction, the composite image signal can be corrected by multiplying it by a gain such that the relative light intensity ratio between pre-prepared fields of view becomes constant. For example, the development unit 311 performs light intensity correction by multiplying the composite image signal of each pixel by a gain that has the characteristic of increasing from the central pixel to the peripheral pixels of the image sensor 302.

[0059] In S604, the developing unit 311 performs noise reduction processing on the composite image signal. As a method for reducing noise, noise reduction using a Gaussian filter can be used.

[0060] In step S605, the development unit 311 performs demosaicing on the composite image signal to generate image data with brightness information for each color (red, green, and blue) for each pixel. As a demosaicing method, a technique can be used in which color information for each pixel is generated using linear interpolation for each color channel.

[0061] In S606, the developing unit 311 performs gradation correction (gamma correction processing) using a predetermined gamma value. The image data Idc(x,y) of pixel (x,y) after gradation correction is expressed by the following equation (Equation 2) using the image data Id(x,y) of pixel (x,y) before gradation correction and the gamma value γ.

[0062]

number

[0063] The gamma value γ can be a pre-defined value. The gamma value γ may also be determined according to the pixel position. For example, the gamma value γ may be changed for each region obtained by dividing the effective area of ​​the image sensor 302 into a predetermined number of divisions.

[0064] In S607, the development unit 311 performs a color space conversion process to convert the color space of the image data from the RGB color space to the YUV color space. The development unit 311 uses predetermined coefficients and a color space conversion formula (Equation 3) to convert the image data corresponding to the luminance of each color (red, green, and blue) into luminance values ​​and color difference values, thereby converting the color space of the image data from the RGB color space to the YUV color space.

[0065] IdcR(x,y) represents the red image data value of pixel (x,y) after gradation correction. IdcG(x,y) represents the green image data value of pixel (x,y) after gradation correction. IdcB(x,y) represents the blue image data value of pixel (x,y) after gradation correction. Y(x,y) represents the luminance value of pixel (x,y) obtained by color space conversion. U(x,y) represents the difference (chrominance value) between the luminance value and the blue component of pixel (x,y) obtained by color space conversion. V(x,y) represents the difference (chrominance value) between the luminance value and the red component of pixel (x,y) obtained by color space conversion. The coefficients (ry,gy,gy) are coefficients for calculating Y(x,y), and the coefficients (ru,gu,gu) and (rv,gv,bv) are coefficients for calculating the chrominance value, respectively.

[0066]

number

[0067] In S608, the developing unit 311 performs a correction (distortion correction) on the converted image data to suppress the effects of distortion aberration caused by the optical characteristics of the imaging optical system 301. The distortion aberration correction process is performed by geometrically transforming the image data to correct the distortion rate of the imaging optical system 301. The geometrical transformation is performed using a polynomial that generates the pixel positions before correction from the correct pixel positions without distortion aberration. If the pixel positions before correction are decimal numbers, they may be rounded to the nearest neighbor pixel, or linear interpolation may be used.

[0068] In S609, the developing unit 311 outputs the image data to the object information generation unit 320, to which distortion correction processing has been applied.

[0069] This completes the development process performed by the development unit 311.

[0070] Furthermore, if the recognition processing unit 321 of the route generation ECU 130, which will be described later, can generate external information through external recognition processing using image data, the development unit 311 does not need to perform all the processes in Figure 6(A). For example, if the recognition processing unit 321 can detect an object within the shooting angle range based on image data to which the distortion correction processing in S608 has not been applied, the process in step S608 may be omitted from the development process in Figure 6(A).

[0071] Figure 6(B) is a flowchart showing the operation of the distance image generation process performed by the distance image generation unit 312. Here, distance image data is data in which distance information corresponding to the distance from the imaging device 110 is associated with each pixel. The distance information may be a distance value D, or it may be a defocus amount ΔL or a parallax amount d used to calculate the distance value. In this embodiment, the distance image data will be described as data to which a distance value D is associated with each pixel.

[0072] In S611, the distance image generation unit 312 generates a luminance image signal from the input image signal. The distance image generation unit 312 generates a first luminance image signal using the first image signal and a second luminance image signal using the second image signal. The distance image generation unit 312 generates a luminance image signal by combining the image signal values ​​of the red, green, and blue pixels of each pixel group 410 using coefficients. Alternatively, the distance image generation unit 312 may generate a luminance image signal by performing demosaicing using linear interpolation and then multiplying each of the red, green, and blue channels by a predetermined coefficient and combining them.

[0073] In S612, the distance image generation unit 312 corrects the light intensity balance between the first luminance image signal and the second luminance image signal. The light intensity balance correction is performed by multiplying at least one of the first luminance image signal and the second luminance image signal by a correction coefficient. The correction coefficient is pre-calculated so that the luminance ratio between the obtained first luminance image signal and the second luminance image signal remains constant after uniform illumination is applied after the position adjustment of the imaging optical system 301 and the image sensor 302, and is stored in the memory 340. The distance image generation unit 312 multiplies at least one of the first luminance image signal and the second luminance image signal by the correction coefficient to generate the first image signal and the second image signal to which the light intensity balance correction has been applied.

[0074] In step S613, the distance image generation unit 312 performs noise reduction processing on the first luminance image signal and the second luminance image signal to which light intensity balance correction has been applied. The distance image generation unit 312 performs noise reduction processing by applying a low-pass filter that reduces high spatial frequency bands to each luminance image signal. Alternatively, the distance image generation unit 312 may use a band-pass filter that transmits a predetermined spatial frequency band. In this case, the effect of reducing the influence of the correction error of the light intensity balance correction performed in step S612 can be obtained.

[0075] In S614, the distance image generation unit 312 calculates the disparity amount, which is the relative positional shift between the first luminance image signal and the second luminance image signal. The distance image generation unit 312 sets a point of interest in the first luminance image corresponding to the first luminance image signal and sets a comparison region centered on the point of interest. Next, the distance image generation unit 312 sets a reference point in the second luminance image corresponding to the second luminance image signal and sets a reference region centered on the reference point.

[0076] The distance image generation unit 312 calculates the correlation between a first luminance image contained within the matching region and a second luminance image contained within the reference region while sequentially moving the reference point, and identifies the reference point with the highest correlation as the corresponding point. The distance image generation unit 312 defines the amount of relative positional shift between the point of interest and the corresponding point as the amount of disparity at the point of interest. By calculating the amount of disparity while sequentially moving the point of interest, the distance image generation unit 312 can calculate the amount of disparity at multiple pixel positions. The distance image generation unit 312 identifies a value indicating the disparity value for each pixel and generates disparity image data, which is data indicating the disparity distribution.

[0077] Furthermore, known methods can be used as the method for calculating the correlation used by the distance image generation unit 312 to determine the amount of disparity. For example, the distance image generation unit 312 can use a method called NCC (Normalized Cross-Correlation) which evaluates the normalized cross-correlation between luminance images. Alternatively, the distance image generation unit 312 may use a method that evaluates the degree of difference as the correlation. For example, the distance image generation unit 312 can use SAD (Sum of Absolute Difference), which evaluates the sum of the absolute differences between luminance images, or SSD (Sum of Squared Difference), which evaluates the sum of the squares of the differences.

[0078] In S615, the distance image generation unit 312 converts the parallax amount of each pixel in the parallax image data into a defocus amount to obtain the defocus amount of each pixel. Based on the parallax amount of each pixel in the parallax image data, the distance image generation unit 312 generates defocus image data that shows the defocus amount of each pixel. The distance image generation unit 312 uses the parallax amount d(x,y) of pixel (x,y) in the parallax image data and the conversion coefficient K to calculate the defocus amount ΔL(x,y) of pixel (x,y) from the following equation (Equation 4). Note that the imaging optical system 301 cuts off a portion of the first light beam 511 and the second light beam 521 in the peripheral field of view due to vignetting. Therefore, the conversion coefficient K is a value that depends on the field of view (pixel position).

[0079]

number

[0080] If the imaging optical system 301 has a characteristic of field curvature in which the focal position changes between the central and peripheral fields of view, then, if the amount of field curvature is Cf, the parallax amount d(x,y) can be converted to the defocus amount ΔL(x,y) using the following equation (Equation 5). After aligning the imaging optical system 301 and the image sensor 302, the relationship between the amount of parallax and the distance to the object can be obtained by chart imaging, thereby obtaining the conversion coefficient K and the amount of field curvature Cf. In this case, the amount of field curvature Cf depends on the field of view and is given as a function of the pixel position.

[0081]

number

[0082] In S616, the distance image generation unit 312 converts the defocus amount ΔL(x,y) of pixel (x,y) to the distance value D(x,y) to the object at pixel (x,y), and generates distance image data. The distance value D to the object can be calculated by converting the defocus amount ΔL using the imaging relationship of the imaging optical system 301. When the focal length of the imaging optical system 301 is f and the distance from the principal point on the image side to the image sensor 302 is Ipp, the defocus amount ΔL(x,y) can be converted to the distance value D(x,y) to the object using the imaging formula shown in the following equation (Equation 6).

[0083]

number

[0084] Note that the focal length f and the distance Ipp from the principal point on the image side to the image sensor 302 are assumed to be constant values ​​regardless of the field of view, but this is not limited to this. If the imaging magnification of the imaging optical system 301 changes significantly with each field of view, at least one of the focal length f or the distance Ipp from the principal point on the image side to the image sensor 302 may be set to a value that changes with each field of view.

[0085] In S617, the distance image generation unit 312 outputs the distance image data to the object information generation unit 320. This completes the distance image data generation process performed by the distance image generation unit 312.

[0086] Furthermore, the parallax amount d, defocus amount ΔL, and distance value D from the principal point of the imaging optical system 301 for each pixel are values ​​that can be converted using the coefficients and conversion formulas described above. Therefore, the distance image data generated by the distance image generation unit 312 may include information representing the parallax amount d or the defocus amount ΔL for each pixel. Considering that the object information generation unit 320 calculates a representative value of the distance value D included in the object region, it is desirable to generate the distance image data based on a defocus amount whose frequency distribution is symmetrical.

[0087] In the process of calculating the disparity amount in S614, corresponding points are searched using the correlation between the first luminance image and the second luminance image. If there is a lot of noise in the first image signal (for example, noise caused by optical shot noise) or if the change in the signal value of the luminance image signal included in the matching region is small, it may not be possible to correctly evaluate the degree of correlation. In such cases, a disparity amount with a large error may be calculated compared to the correct disparity amount. If the error in the disparity amount is large, the error in the distance value D generated in S616 will also be large.

[0088] The distance image generation process performed by the distance image generation unit 312 may include a reliability calculation process for calculating the reliability of the disparity amount (disparity reliability). Disparity reliability is an index that indicates how much error is contained in the calculated disparity amount. For example, the ratio of the standard deviation to the mean value of the signal values ​​included in the matching area can be evaluated as the disparity reliability. When the change in signal values ​​within the matching area (so-called contrast) is large, the standard deviation becomes large. When the amount of light incident on the pixel is large, the mean value becomes large. When the amount of light incident on the pixel is large, there is a lot of light shot noise. In other words, the mean value has a positive correlation with the amount of noise.

[0089] The ratio of the mean to the standard deviation (standard deviation / mean) corresponds to the ratio of the contrast magnitude to the noise level. If the contrast is sufficiently large relative to the noise level, it can be estimated that the error in calculating the parallax amount is small. In other words, the higher the parallax confidence, the smaller the error in the calculated parallax amount, and the more accurate the parallax amount can be said to be.

[0090] In S614, the parallax confidence level is calculated at each point of interest, and confidence data representing the likelihood of the distance value for each pixel constituting the distance image data can be generated. The distance image generation unit 312 can output the confidence data to the object information generation unit 320.

[0091] Next, we will describe the process by which the object information generation unit 320 generates external information and object distance information based on image data and distance image data.

[0092] Figure 7 is a flowchart showing the processes performed by the object information generation unit 320. Figure 8 is a schematic diagram showing the images and information in the processes performed by the object information generation unit 320.

[0093] The recognition processing unit 321 generates external information based on the image data, including the position of an object in the image, its size (such as width and height), area information, the type of object (attributes), and an identification number (ID number). The identification number is identification information for identifying the detected object, and is not limited to a number. The recognition processing unit 321 detects the type of object present within the imaging angle of the imaging device 110, as well as the position and size of the object in the image, and determines whether or not the object has already been registered, and assigns an identification number.

[0094] Figure 8(A) shows an image 810 based on image data acquired by the imaging device 111 and input to the object information generation unit 320. Image 810 includes a person 801, a vehicle 802, a sign 803, a road 804, and a lane 805. The recognition processing unit 321 detects objects from image 810 and generates external information indicating the type, identification number, and region information of each object. Figure 8(B) is a schematic diagram showing the external information of objects detected from image 810 on the xy coordinate plane, at their positions in image 810. For example, the external information is generated as a table as shown in Table 1. In the external information, the region of an object is defined as a rectangular frame (object frame) surrounding the object. In the external information, the region information of an object is indicated by the shape of the rectangular object frame as the coordinates of the top left (x0, y0) and the bottom right (x1, y1).

[0095] [Table 1]

[0096] Figure 7(A) is a flowchart showing the process by which the recognition processing unit 321 generates external information. The recognition processing unit 321 starts the process of generating external information in response to the acquisition of image data.

[0097] In S701, the recognition processing unit 321 generates image data to be used for object detection processing from the image data. The recognition processing unit 321 then processes the image data input from the image processing unit 310 to expand or contract its size to a size determined by the detection performance and processing time in the object detection process.

[0098] In S702, the recognition processing unit 321 performs a process to detect objects contained in the image based on the image data, and detects the region corresponding to the object and the type of object in the image. The recognition processing unit 321 may detect multiple objects from a single image. In this case, the recognition processing unit 321 identifies the type and region for the multiple detected objects.

[0099] The recognition processing unit 321 generates external information including the position and size (horizontal width, vertical height) of the area in the image where the object is detected, and the type of object. The types of objects that the recognition processing unit 321 can detect include, for example, vehicles (passenger cars, buses, trucks), people, motorcycles, signs, etc. The recognition processing unit 321 detects objects and identifies the type of detected object by comparing the outline of the object in the image with a predetermined outline pattern associated with the type of object. The types of objects that the recognition processing unit 321 can detect are not limited to those mentioned above, but from the viewpoint of processing speed, it is desirable to narrow down the number of object types to be detected according to the driving environment of the vehicle 100.

[0100] In S703, the recognition processing unit 321 tracks objects for which an identification number has already been registered. The recognition processing unit 321 identifies objects for which an identification number has already been registered among the objects detected in S702. An object with a registered identification number is, for example, an object that was detected in a previous object detection process and assigned an identification number. When an object with a registered identification number is detected, the recognition processing unit 321 associates the object type and region information acquired in S702 with the external information corresponding to that identification number (updates the external information).

[0101] If it is determined that an object with a registered identification number does not exist in the image information, the tracking is interrupted, as it is determined that the object associated with that identification number has moved outside the field of view of the camera 110 (is lost).

[0102] In S704, the recognition processing unit 321 determines for each object detected in S702 whether it is a new object for which an identification number has not been registered. If an object is determined to be new, a new identification number is assigned to the external information indicating the type and region of that object, and registered in the external information.

[0103] In step S705, the recognition processing unit 321 outputs the generated external information, along with time information, to the route generation device 150.

[0104] This completes the process of generating external information performed by the recognition processing unit 321.

[0105] Figure 7(B) is a flowchart showing the process by which the distance information generation unit 322 generates distance information for each object. Based on external information and distance image data, the distance information generation unit 322 generates object distance information that represents the distance value for each detected object.

[0106] Figure 8(C) shows image 810 based on the image data of Figure 8(A) and a distance image 820 based on the corresponding distance image data. In the distance image 820, distance information is indicated by the intensity of the color, with darker colors representing closer objects and lighter colors representing farther objects.

[0107] In S711, the distance information generation unit 322 counts the number of objects Nmax detected by the recognition processing unit 321 and calculates the number of detected objects.

[0108] In S712, the distance information generation unit 322 sets N to 1 (initialization process). The processes from S713 onward are executed sequentially for each object shown in the external information. Assume that the processes from S713 to S716 are executed in order of increasing identification number in the external information.

[0109] In S713, the distance information generation unit 322 identifies a rectangular region on the distance image 820 that corresponds to the region (object frame) on the image 810 of the Nth object included in the external information. The distance information generation unit 322 sets a frame (object frame) that shows the outline of the corresponding region on the distance image 820. Figure 8(D) is a schematic diagram in which a frame showing the outline of the region set on the distance image 820 is superimposed for each object detected from the image 810. As shown in Figure 8(D), the distance information generation unit 322 sets an object frame 821 corresponding to a person 801, an object frame 822 corresponding to a vehicle 802, and an object frame 823 in the case of a sign 803 on the distance image 820.

[0110] In S714, the distance information generation unit 322 generates a frequency distribution of distance information for pixels contained within the rectangular region of the distance image 820 corresponding to the Nth object. When the information associated with each pixel of the distance image data is the distance value D, the intervals of the frequency distribution are set so that the reciprocals of the distances are equally spaced. If the defocus amount or disparity amount is associated with each pixel of the distance image data, it is desirable to divide the intervals of the frequency distribution into equal intervals.

[0111] In S715, the distance information generation unit 322 uses the most frequently occurring distance information from the frequency distribution as object distance information indicating the distance of the Nth object.

[0112] Alternatively, the average of the distance values ​​contained within the region may be calculated and used as object distance information. When calculating the average, a weighted average can be used using confidence data. By setting a larger weight for each pixel as the confidence level of the distance value increases, the object distance can be calculated with greater accuracy.

[0113] Furthermore, it is desirable that the object distance information be information indicating the distance from a predetermined position on the vehicle 100 to the object, in order to facilitate path generation in the path generation process described later. When using a distance value D as the distance information, since the distance value D indicates the distance from the image sensor 302 to the object, the most frequent value can be offset by a predetermined amount to obtain information indicating the distance from a predetermined position on the vehicle 100 to the object. When using a defocus amount ΔL as the distance information, it is converted to the distance from the image sensor 302 using equation 6, and then offset by a predetermined amount to obtain information indicating the distance from a predetermined position on the vehicle 100 to the object.

[0114] In S716, the distance information generation unit 322 determines whether N is less than the number of detected objects Nmax+1. If N is less than the number of detected objects Nmax+1 (S716 Yes), in S717, the distance information generation unit 322 sets N to N+1, and the process returns to S713. That is, object distance information is extracted for the next object (the N+1th object). Otherwise, if N is greater than or equal to the number of detected objects Nmax+1 (S716 No), the process terminates.

[0115] In S718, the distance information generation unit 322 outputs object distance information for N objects along with time information to the path generation unit 330, and the process ends.

[0116] Through the object distance information generation process described above, object distance information is generated for each object included in the external information. In particular, by statistically determining the object distance information from the distance information contained in the region of the distance image 820 corresponding to the object detected in image 810, it is possible to suppress variations in pixel-by-pixel distance information caused by noise and calculation accuracy. Therefore, it is possible to obtain information indicating the distance of objects with higher accuracy. The method for statistically determining distance information is to take the most frequent distance information, mean, median, etc., from the distribution of distance information, and various methods can be employed.

[0117] Next, we will explain the process (route generation process) that generates route information, which is performed by the route generation unit 330 of the route generation ECU 130. Route information includes information such as the direction of travel and speed of the vehicle. Route information can also be said to be operation plan information. The route generation unit 330 outputs the route information to the vehicle control ECU 140. The vehicle control ECU 140 controls the drive unit 170 based on the route information.

[0118] In this embodiment, the path generation unit 330 generates path information so that the vehicle 100 follows the vehicle in front when there is a vehicle (a vehicle ahead) in the direction of travel of the vehicle 100. Furthermore, the path generation unit 330 generates path information so that the vehicle 100 takes evasive action to avoid colliding with an object.

[0119] Figure 9 is a flowchart illustrating the route generation process performed by the route generation unit 330. Figure 9(A) is a flowchart illustrating the operation of the route generation process performed by the route generation unit 330. The route generation unit 330 generates route information for the vehicle 100 based on external information, object distance information, and distance information generated by the radar device 120. The route generation ECU 130 reads the external information, object distance information, and distance information generated by the radar device 120 for each time point from the memory 340 provided in the route generation device 150 and processes it.

[0120] In S901, the path generation unit 330 detects objects on the planned driving path of the vehicle 100 from external information and object distance information. The path generation unit 330 determines objects on the driving path by comparing the direction of travel planned by the vehicle 100 with the position and type of objects included in the external information. The direction of travel planned by the vehicle 100 is determined based on information regarding the braking of the vehicle 100 (steering angle, speed, etc.) obtained from the vehicle control ECU 140. If no objects are detected on the driving path, the path generation unit 330 determines "no objects".

[0121] Assume that the image 810 shown in Figure 8(A) was acquired by the imaging device 110. If the path generation unit 330 determines from the braking information of the vehicle 100 acquired from the vehicle control ECU 140 that the vehicle 100 is moving in the direction along lane 805, the path generation unit 330 detects the vehicle 802 as an object on the travel path.

[0122] In S902 and S903, the path generation unit 330 determines whether to generate path information for follow-the-object driving or path information for avoidance driving, based on the distance between the vehicle 100 and the object on the driving path and the speed Vc of the vehicle 100.

[0123] In S902, the path generation unit 330 determines whether the distance between the vehicle 100 and an object on the travel path is shorter than the threshold Dth. The threshold Dth is expressed as a function of the vehicle's travel speed Vc. The higher the travel speed, the larger the threshold Dth. If the path generation unit 330 determines that the distance between the vehicle 100 and an object on the travel path is shorter than the threshold Dth (S902 Yes), the process proceeds to S903. If the path generation unit 330 determines that the distance between the vehicle 100 and an object on the travel path is greater than or equal to the threshold Dth (S902 No), the process proceeds to S908.

[0124] In S903, the route generation unit 330 determines whether the relative speed between the vehicle 100 and the object on the route is a positive value. The route generation unit 330 obtains the identification number of the object on the route from external information and obtains object distance information for the object on the route at each time from external information obtained from the present time up to a predetermined time ago. The route generation unit 330 calculates the relative speed between the vehicle 100 and the object on the route from the object distance information for the period up to the predetermined time ago that has been obtained. If the relative speed is a positive value, it indicates that the vehicle 100 and the object on the route are approaching each other.

[0125] If the path generation unit 330 determines that the relative speed between the vehicle 100 and the object on the travel path is a positive value (S903 Yes), the process proceeds to S904. If the path generation unit 330 determines that the relative speed between the vehicle 100 and the object on the travel path is not a positive value (S903 No), the process proceeds to S908.

[0126] If the process proceeds to S904, route information for executing evasive action is generated. If the process proceeds to S908, route information for executing follow-me driving is generated.

[0127] In other words, based on the determinations in S902 and S903, the path generation unit 330 determines to perform an evasive action if the distance between the vehicle 100 and the object on the travel path is shorter than the threshold Dth, and the relative speed between the vehicle 100 and the object on the travel path is a positive value. The path generation unit 330 determines to follow the object if the distance between the vehicle 100 and the object on the travel path is greater than or equal to the threshold Dth. Alternatively, the path generation unit 330 determines to follow the object if the distance between the vehicle 100 and the object on the travel path is shorter than the threshold Dth, and the relative speed between the vehicle 100 and the object on the travel path is a negative value.

[0128] If the distance between vehicle 100 and an object on the travel path is shorter than a threshold Dth calculated from the speed of vehicle 100, and the relative speed between vehicle 100 and the object on the travel path is a positive value, then there is a high probability that vehicle 100 will collide with the object on the travel path. Therefore, the path generation unit 330 generates path information to take evasive action. Otherwise, the path generation unit 330 takes a follow-the-object driving stance.

[0129] Furthermore, the above determination may include a determination of whether the detected object on the travel path is a moving object (such as a passenger car or other vehicle).

[0130] In S904, the path generation unit 330 starts the process of generating path information for executing evasive action.

[0131] In S905, the path generation unit 330 acquires information regarding avoidance space. The path generation unit 330 acquires distance information indicating the distance from the radar device 120 to objects to the sides and rear of the vehicle 100. Based on the distance information acquired from the radar device 120, the speed of the vehicle 100, and information indicating the size of the vehicle 100, the path generation unit 330 acquires information indicating the direction and size of the space around the vehicle 100 in which the vehicle 100 can move.

[0132] In S906, the path generation unit 330 sets path information for avoidance action based on information indicating the direction and size of the space in which the vehicle 100 can move, external information, and object distance information. The path information for avoidance action is, for example, information to move the vehicle 100 to the right while decelerating if there is space to the right of the vehicle 100.

[0133] In S907, the route generation unit 330 outputs route information to the vehicle control ECU 140. The vehicle control ECU 140 determines parameters to control the drive unit 170 based on the route information so that the vehicle 100 travels along the route indicated in the acquired route information, and controls the drive unit 170. Specifically, the vehicle control ECU 140 determines the steering angle, accelerator control value, brake control value, control signal for gear engagement, and lamp illumination control signal based on the route information.

[0134] In S908, the route generation unit 330 starts the process of generating route information for performing follow-up operation.

[0135] In S909, the route generation unit 330 generates route information so that the vehicle 100 follows an object (a preceding vehicle) on its travel path. Specifically, the route generation unit 330 generates route information so that the distance between the vehicle 100 and the preceding vehicle (inter-vehicle distance) remains within a predetermined range. For example, if the relative speed between the vehicle 100 and the preceding vehicle is a positive value and the inter-vehicle distance is shorter than the predetermined range, the route generation unit 330 generates route information so that the vehicle 100 decelerates while maintaining a straight direction of travel.

[0136] The route generation unit 330 generates route information so that the vehicle 100's speed does not exceed a predetermined value (for example, the legal speed limit on the road the vehicle 100 is traveling on, or a set speed based on instructions from the driver 101). The process then proceeds to S907, where the vehicle control ECU 140 controls the drive unit 170 based on the generated route information.

[0137] With the above steps completed, the route information generation process by the route generation unit 330 is finished. The route information generation process is assumed to be executed repeatedly while the vehicle 100 is in motion.

[0138] According to the control described above, by integrating distance information within a frame based on the position and size of objects in the image, the effects of sensor noise and local distance errors caused by high-luminance reflections from the subject are reduced, and the distance value to each object can be calculated with high accuracy. This increased accuracy in distance values ​​to each object allows the route generation ECU 130 to calculate the vehicle 100's route with greater precision, enabling the vehicle 100 to travel more stably.

[0139] <Example 1> In the process described above, the object's region was shown as a rectangular area (object frame) encompassing the object, but the object's region may also be defined as an area bounded by the outer perimeter of the object within the image. In S702, the recognition processing unit 321 stores the region in the image 810 where the object exists as the object's region in the external information. For example, the recognition processing unit 321 can divide the image into regions for each object by identifying attributes for each pixel of the image data.

[0140] Figure 8(E) is a schematic diagram showing the results of the recognition processing unit 321 performing region division for each object superimposed on the image information 810. Region 831 represents the region of person 801, region 832 represents the region of vehicle 802, and region 833 represents the region of sign 803. Furthermore, region 834 represents the region of road 804, and region 835 represents the region of lane 805.

[0141] In this case, in S713 and S714, the distance information generation unit 322 calculates the frequency distribution of distance values ​​contained within each region shown in Figure 8(E) for each object.

[0142] By defining the object's region in this way, distance information from non-object elements such as the background becomes less likely to be included within the region. In other words, the distribution of distance information within the region becomes more accurately reflected in the object's distance information. Therefore, the influence of non-object elements such as the background and foreground can be reduced, allowing for more accurate calculation of the object's distance value.

[0143] <Modification 2> In this embodiment, the imaging device 110 sequentially outputs image information and distance image information. Furthermore, the recognition processing unit 321 sequentially generates external information using the sequentially received image information. The external information includes the identification number of an object, and if an object with the same identification number is detected at a certain time T0 and time T1, it is possible to determine the time changes in the distance information and detected size of that object. In the modified example 2, the distance information generation unit 322 calculates the average of the distance values ​​D of objects with the same identification number within a predetermined time range. This reduces distance variation in the time direction.

[0144] Figure 10 is a flowchart showing the object distance information generation process performed by the distance information generation unit 322 in the modified example 2. For each process in this flowchart, the processes indicated with the same numbers as those shown in Figure 7(B) are the same as the processes described above, so their explanation is omitted.

[0145] In S721, the distance information generation unit 322 uses the most frequently occurring distance information from the frequency distribution as object distance information indicating the distance of the Nth object. The distance information generation unit 322 then stores the object distance information, along with the identification number and time, in the memory 340.

[0146] In S722, the distance information generation unit 322 retrieves the history of object distance information with the same identification number as the Nth object from the object distance information stored in the memory 340. The distance information generation unit 322 retrieves object distance information with the same identification number corresponding to a predetermined time before the time corresponding to the latest object distance information. Figure 11 is a schematic diagram showing the time change of object distance information for objects with the same identification number as the Nth object. The horizontal axis shows time, and the vertical axis shows object distance information (distance value D). Time t0 indicates the time when the latest distance value D was acquired.

[0147] In S723, the distance information generation unit 322 calculates the average value of object distance information included in a time range from the time the latest object distance information was acquired to a predetermined time before, based on the history of object distance information of objects with the same identification number as the acquired Nth object. For example, in Figure 11, the distance information generation unit 322 calculates the average value of the distance values ​​of four points included in a predetermined time range ΔT.

[0148] As described above, by using the identification number included in the external information to obtain a history of object distance information (distance value) for the same object and taking a time average, it is possible to suppress variability. Even when the road on which the vehicle 100 travels changes (for example, curves, slopes, rough roads with many bumps, etc.), it is possible to track the same object and calculate the average value in the time direction. Therefore, it is possible to reduce the impact of changes in the driving environment, reduce variability in distance values ​​due to noise such as optical shot noise included in the image signal, and calculate the distance value of the object with greater accuracy.

[0149] <Variation 3> In the above-described modification 2, variability was suppressed by averaging the object distance information of objects with the same identification number over time. When averaging the history of object distance information within a predetermined time range, the number of samples used for averaging can be increased by extending the time range. Therefore, the variability of the distance value from vehicle 100 to the object can be further reduced. However, if the distance from vehicle 100 to the object changes within the predetermined time range, the averaging will include the change in distance, which may prevent the accurate estimation of the distance value from vehicle 100 to the object. In modification 3, by performing a weighted averaging of object distance information using the size of objects with the same identification number, it becomes possible to obtain the distance between vehicle 100 and the object with higher accuracy.

[0150] Figure 12 is a flowchart showing the object distance information generation process performed by the distance information generation unit 322 in the modified example 3. For each process in this flowchart, the processes indicated with the same numbers as those shown in Figure 7(B) are the same as the processes described above, so their explanation is omitted.

[0151] In S721, the distance information generation unit 322 uses the most frequently occurring distance information from the frequency distribution as object distance information indicating the distance of the Nth object. The distance information generation unit 322 then stores the object distance information, along with the identification number and time, in the memory 340.

[0152] In S731, the distance information generation unit 322 retrieves from memory 340 the history of object distance information and the history of information indicating the size of the object for the Nth object and the object with the same identification number. The information indicating the size of the object is obtained from the information indicating the object frame stored in the external information. For example, the width (x1-x0) from the top-left coordinate (x0, y0) and bottom-right coordinate (x1, y1) is used as information indicating the size of the object.

[0153] In S732, the distance information generation unit 322 uses information indicating the size of objects with the same identification number as the Nth object to perform a weighted averaging process on object distance information for the same identification number corresponding to a predetermined time period prior to the time corresponding to the latest object distance information. The weight coefficient for each time period is determined using the object size at the corresponding time period.

[0154] Figure 13 is a schematic diagram illustrating weighted averaging. Figure 13(A) is a schematic diagram showing image 1300 based on image data acquired by the imaging device 110 at time t1, prior to time t0, which corresponds to the latest object distance information. Image 1300 includes vehicle 1301. Frame 1302 shows the object frame of vehicle 1301 determined from image 1300.

[0155] Figure 13(B) is a schematic diagram showing image 1310 based on image data acquired by the imaging device 110 at time t0. Image 1310 includes the vehicle 1301, similar to image 1300. Frame 1312 indicates the object frame corresponding to the vehicle 1301 in image 1310. At time t0, the size of the vehicle 1301 in image 1300 is larger than the size in image 1310 acquired at time t1. Object frame 1312 is larger than object frame 1302.

[0156] Since the size of an object in an image is proportional to the horizontal magnification of the imaging optical system 301, the distance between the object and the vehicle 100 is directly proportional to the reciprocal of the object's size in the image information. By comparing the object's size in the image information at different time points, for example, an increase in size indicates that the distance between the object and the vehicle 100 has decreased, while a decrease in size indicates that the distance between the object and the vehicle 100 has increased. Furthermore, a small change in size indicates a small change in the distance between the object and the vehicle 100.

[0157] As an example, let's assume that we obtain distance information for vehicle 1301 as the Nth object.

[0158] Figure 13(C) is a schematic diagram showing the time change of object distance information for an object with the same identification number as vehicle 901 (the Nth object). Figure 13(D) is a schematic diagram showing the time change of the reciprocal of the size information (width) for an object with the same identification number as vehicle 901 (the Nth object).

[0159] In S732, the distance information generation unit 322 compares the reciprocal of the object's size (width) at time t0 with the reciprocal of the object's size (width) at each time point from time t0 to a predetermined time prior. The distance information generation unit 322 determines the weighting coefficient such that the larger the absolute value of the difference in the reciprocals of the object's size (width), the smaller the weighting coefficient becomes. Note that the relationship between the reciprocal of the object's size (width) and the weighting coefficient is not limited to the example above; for example, the weighting coefficient may be determined according to the ratio of the reciprocal of the object's size (width) at each time point to the reciprocal of the object's size (width) at time t0.

[0160] In S732, the distance information generation unit 322 weights and averages the object distance information using weighting coefficients to obtain the object distance information of the vehicle 1301 at time t0.

[0161] According to the processing in Modification 3, by weighting and averaging object distance information using weight coefficients determined using the size of the object in the image, the estimation error of distance values ​​caused by relative distance changes from the vehicle 100 to the object can be reduced.

[0162] <Second Embodiment> In the first embodiment, the distance to an object in an image (object distance information) was obtained with high accuracy by statistically processing distance data acquired for the object using the image plane phase-difference method. In the second embodiment, the distance to an object can be obtained with even higher accuracy by combining the image plane phase-difference method for calculating distance and the image recognition method for calculating distance. Hereafter, the distance calculation method and the distance calculation process based on that method will be referred to as "distance measurement".

[0163] A second embodiment of the present invention will be described in detail below with reference to the figures. The following processes can be performed by any or a combination of the image processing unit 310 and object information generation unit 320 of the imaging device 110, and the path generation ECU 130. In the following description, it will be assumed that the distance information generation unit 322 of the object information generation unit 320 performs the following processes. However, the application of the present invention is not limited to this.

[0164] Furthermore, in explanations that refer to diagrams, the same reference numeral will be used for parts that represent the same area, even if the diagram numbers are different, and redundant explanations will be omitted.

[0165] In the second embodiment, the path generation device 150 corrects the measured distance value by combining imaging plane phase difference distance measurement and distance measurement using image recognition.

[0166] Image plane phase difference distance measurement is a distance measurement method using the image plane phase difference method described in the first embodiment.

[0167] Distance measurement using image recognition includes distance measurement that calculates the distance from the width of an object detected based on object recognition (object width distance measurement) and distance measurement that calculates the distance from the ground contact point of an object (ground contact point distance measurement).

[0168] Object width distance measurement calculates the distance to an object by utilizing the principle that the fewer pixels an object has in its width on an image, the farther the object is, and the more pixels an object has in its width, the closer the object is. Other parameters indicating the size of an object on an image, such as height and diagonal direction, can also be used in the distance calculation.

[0169] In ground contact distance measurement, it is assumed that the object is in contact with, for example, the road surface, and the distance from vehicle 100 to the object is calculated based on the distance between the ground contact line on the object's image (which would be the bottom edge if the road surface is below in the image) and the vanishing point on the image. The closer the ground contact line is to the vanishing point, the greater the distance the object is from vehicle 100, and the farther the ground contact line is from the vanishing point, the closer the object is from vehicle 100.

[0170] This section explains the characteristics of errors in image plane phase difference distance measurement, object width distance measurement, and ground contact position distance measurement. Common to all of these methods, relative error and absolute error are defined as follows: Relative error is defined as the quantity equivalent to the standard deviation for a sufficient number of samples when there is no change in relative distance. Absolute error is defined as the quantity equivalent to the difference between the mean value and the true value for a sufficient number of samples when there is no change in relative distance.

[0171] The relative error in image-plane phase-difference ranging is primarily caused by parallax errors resulting from block matching due to pixel value variations caused by sensor noise. Since this does not change depending on the parallax value, when converted to distance, the relative error basically worsens in proportion to the square of the distance. Absolute errors arise from optical system aberrations, assembly errors, and fluctuations due to heat and vibration. While it is possible to correct for each of these factors, if correction is not performed considering the computational load, a significant amount of error may remain.

[0172] The relative error in object width measurement depends on the resolution and recognition accuracy of the object in the image. In object width measurement, the width of the detected object in the image cannot be converted to distance unless the actual object width (the actual width of the object, expressed in units such as meters) is known. Therefore, the actual object width must be determined in some way, and both the absolute and relative errors depend on that actual object width. Since the relative error is proportional to the distance, the relative error may be smaller than that of image-plane phase-difference distance measurement at long distances.

[0173] The relative error in ground contact distance measurement depends on the accuracy of recognizing the ground contact line on the object's image and the image resolution. Since image resolution is the resolution used to measure the distance between the vanishing point and the ground contact line, higher resolution images allow for more accurate measurements even at long distances. Furthermore, when the road surface extends downwards, the estimation error of the pitch angle in the optical axis direction becomes the distance measurement error.

[0174] When an image acquisition device is attached to a moving object, the pitch angle fluctuates with each image frame due to the acceleration of the movement and the road surface conditions. In this case, the pitch angle error is a relative error. A pitch angle error that always occurs at a constant value due to the installation conditions and the tilt of the moving object itself is an absolute error. As will be discussed later, the pitch angle can be estimated using vanishing point information and movement information to reduce the error. The relative error is proportional to the distance and is equivalent to the width of the object, but the amount of error is larger than that of the width of the object because it is affected by the pitch angle estimation. Distance measurement based on the object's ground contact position requires that the object is in contact with the ground, so there is a problem in that distance cannot be measured if the lower end is not in contact with the ground, such as with traffic signals or signs.

[0175] Figure 14 is a flowchart showing the process for acquiring distance measurement values ​​(object distance information) in the second embodiment.

[0176] Data D1401 is data input to the distance information generation unit 322. As described in the first embodiment, the distance information generation unit 322 receives external information about an object obtained by image recognition from the image captured by the image sensor 302 (object identification number, object type (attributes), and size of the area corresponding to the object). Data D1401 may also include results from other image recognition processes, and any image recognition result information that shows the pixel position on the image is acceptable, such as image coordinates representing the image range of the recognized image or information about the object's area obtained by semantic region segmentation technology.

[0177] Furthermore, the distance information generation unit 322 receives distance image data as data D1401, which represents the results (distance information for each pixel) obtained by calculation using the image plane phase difference method. In this embodiment, in addition to this data, information regarding the focal length f of the imaging optical system 301, the moving speed of the vehicle 100, and the installation position of the imaging device 110 are also input to the distance information generation unit 322 as data D1401. Data D1401 is a data set that combines all of this information.

[0178] In the following steps, recognition processing may be performed on multiple objects simultaneously, but the processing that involves saving and referencing time-series data information will be performed only on objects that have been recognized as the same object. In other words, they have the same identification number as input data.

[0179] In S1401, the distance information generation unit 322 acquires a measured distance D1 by ground contact position distance measurement and outputs it as data D1402. The measured distance D1 indicates the distance between the vehicle 100 (imaging device 101) and the target object calculated by ground contact position distance measurement. Data D1402 is information indicating the measured distance D1 calculated in S1401 by ground contact position distance measurement.

[0180] The distance information generation unit 322 obtains the position of the pixel in contact with the ground on the image using image coordinates that represent the image range of the recognized image contained in the data D1401. The outline of the distance measurement process when the optical axis is set to be parallel to the road surface and at height H, and the distance between the vanishing point on the image and the ground line is Hs pixels (or subpixel units) is described below. If the image is in the central projection method (or an image corrected to the central projection method) with a focal length f and the pixel size is Ps, the measured distance value (distance) D1 can be expressed as follows using the following equation (Equation 7).

[0181]

number

[0182] Even in cases where the surface is not a road, the contact surface and the optical axis are not parallel, the method is not central projection, or there is significant distortion, distance measurement calculations are still possible if the vanishing point and the contact line can be assumed.

[0183] Equation 7 assumes that the line is parallel to the road surface. However, as mentioned earlier, if there is an error in the pitch angle of the moving object, the vanishing point will be in a different position than assumed, resulting in an error in the value of Hs and consequently a distance error. Furthermore, if the recognition accuracy is poor, Hs will be recognized as being in a different position from the actual grounding line, resulting in a similar distance error.

[0184] In S1403, the distance information generation unit 322 acquires the distance value D3 by imaging plane phase difference distance measurement and outputs it as data D1404. Data D1404 is information indicating the distance value D3 calculated in S1403 by imaging plane phase difference distance measurement.

[0185] As shown in the first embodiment, the distance information generation unit 322 can acquire distance information (distance value) of a target object based on distance image data and external information. For example, suppose that the input data D1401 is distance image data in which the distance information of each pixel is shown as a defocus amount, and external information in which the area of ​​the object is shown as a frame. At this time, the distance information generation unit 322 can acquire the distance between the vehicle 100 and the target object (object distance information) from the imaging formula using the most frequent value of the defocus amount included in the object frame of the target object and the focal length f. The obtained distance is acquired as the measured distance value D3. Note that the input data D1401 may be the distance value itself, or it may be other data in the process of calculation.

[0186] In S1402, the distance information generation unit 322 obtains the width (object width) Ws of the target object in the image. The object width Ws is expressed in terms of the number of pixels. The number of objects may also be expressed in subpixel units. The distance information generation unit 322 measures the object width Ws from the image recognition result. For example, if the external information included in data D1401 includes information indicating the object frame of each object in the image, the width of the object frame corresponding to the target object may be used as the object width Ws. It does not have to be the width of the object; it may also be the height, and either with a larger number of pixels may be chosen, or both may be used to improve robustness. The distance information generation unit 322 sets the information indicating the object width Ws as m data D1403.

[0187] In S1404, the distance image generation unit 322 calculates the actual object width W using the object width Ws from data D1403 and either the distance value D1 obtained by ground position distance measurement from data D1402 or the distance value D3 obtained by imaging plane phase difference distance measurement from data D1404, or both. The actual object width W is information that expresses the width of the target object in units of length (such as meters). The actual object width W can be determined using either or both of the distance value D1 and the distance value D3, but it is desirable to select the distance value with the smallest absolute error. When using the distance value D1, the actual object width W can be expressed as follows using the following equation (Equation 8).

[0188]

number

[0189] Furthermore, the actual object width W may be determined based on information indicating the type of object included in the external information of the input data D1401. For example, if the type of object is a passenger car, the actual object width W can be set to a predetermined value of 1.7m. However, if the actual object width W is determined according to the type of object, it will strictly differ for each object, and the difference will be an absolute error. The distance information generation unit 322 outputs information indicating the actual object width W as data D1405.

[0190] In S1405, the distance information generation unit 322 obtains a distance value D2 using the actual object width W and the object width Ws. Data D1406 is information indicating the distance value D2 calculated in step S1405 based on the object width. The process performed in S1405 is the reverse of the process in step S1404. The distance value D2 can be expressed as follows using the following equation (equation 9).

[0191]

number

[0192] Here, since the actual object width W is the same, D1 = D2. However, because S1404 has additional processing using the time-series information described later, D1 and D2 will have different distance values.

[0193] Step group C1101, consisting of S1402, S1404, and S1405, is for measuring the width of an object.

[0194] In S1406, the distance information generation unit 322 integrates the measured distance D1 from data D1402, the measured distance D2 from data D1406, and the measured distance D3 from data D1404 to obtain the distance value D to the recognized object.

[0195] The integration process is, for example, the process of selecting one of the distance values ​​D1, D2, and D3 as the distance value D. Distance values ​​D1, D2, and D3 each have different relative and absolute errors depending on the type of distance measurement obtained. By selecting the distance value that is thought to have the smallest relative and absolute errors, the distance information generation unit 322 can adopt the distance value with the smallest error from among multiple distance measurement methods, depending on the situation. For example, distance values ​​obtained by ground contact position distance measurement or image plane phase difference distance measurement have smaller errors as the distance value increases. Therefore, if the acquired distance value is greater than (farther than) a predetermined distance, the distance information generation unit 322 selects either distance value D1 or D2, and if it is less than or equal to (closer to) a predetermined distance, it selects distance value D3.

[0196] Another integrated processing method involves considering absolute and relative errors for each, calculating the probability distribution of existence for each distance, and selecting the one that maximizes the probability of existence from the sum of the probability distributions. Furthermore, based on the movement speed, accelerator, brake, and steering information of vehicle 100 included in data D1401, the probability of existence of the relative distance value can be determined for the current relative distance value. For example, to prevent large changes in acceleration, the probability can be set so that the same acceleration as the previous time point is the maximum probability, and the probability decreases as the acceleration change increases. The probability of existence of the relative distance can be calculated accordingly. If accelerator information is available, the maximum probability can be determined in the direction of increasing acceleration, and if brake information is available, the maximum probability can be determined in the direction of decreasing acceleration. Furthermore, this can be determined depending on the type of object. If the type of object is a passenger car or motorcycle, the recognized object may also accelerate and decelerate significantly, so the change in relative distance may be large. If the category is pedestrian, etc., and there is no sudden acceleration or deceleration, the change in relative distance is likely to depend on the user's own actions, and the probability of existence can be determined with higher accuracy.

[0197] The above explains the basic flow when a single input data point at a specific time is received. Next, we will explain the case when a series of time-series input data points are received.

[0198] Figure 15 is a flowchart showing the processing when multiple time-series data points are input. The processing described above using Figure 14, and the same processing and data as in Figure 14, are used, and the same codes are assigned to the data, so the explanation is omitted.

[0199] When sequential input data is received in a time series, for objects assigned the same identification number, i.e., objects recognized as the same object, the distance measurement value D1 of D1402, the object width Ws of D1403, and the distance measurement value of D1404 are used. D3 These can be obtained sequentially in chronological order.

[0200] When continuous time-series input data is received, each distance measurement value changes over time due to the change in the relative distance to the target object. However, assuming the object is a rigid body, the actual object width W can be said to be always constant. Therefore, in S1504, the distance information generation unit 322 smooths the actual object width W in the time-series direction even if the relative distance value changes over time, and obtains the average actual object width W'. This makes it possible to reduce relative errors.

[0201] When W is assumed in equation 8, sufficient smoothing in the time series direction leaves an absolute error equivalent to the ground contact distance measurement value, but the relative error can be made sufficiently small, including the ground contact distance measurement value and object width (number of pixels). The same consideration can be made when the actual object width W' is calculated using the image plane phase difference distance measurement value. Data D1505 becomes the object width W' with small relative error, and the distance conversion process in step S1405 results in the data D1406 Object width distance value D2 This is obtained as a distance value where the absolute error is equivalent to the ground contact position distance measurement value, with only the relative error of the object width Ws.

[0202] In S1507, the distance information generation unit 322 calculates the correction amount for the absolute error of the image plane phase difference distance measurement value D2. The absolute error of the image plane phase difference distance measurement value mainly consists of a component that is a constant value regardless of distance when converted to defocus. Therefore, the distance information generation unit 322 calculates the correction amount for the absolute error of the image plane phase difference distance measurement value D3 The defocus amount is determined based on the focal length and imaging formula. Similarly, the distance information generation unit 322 generates the object width distance value. D2 This is converted to a defocus amount using the equivalent focal length and imaging formula. This may also be done by converting to a defocus amount using the ground contact position distance value D1 or the relative distance value D. The distance information generation unit 322 converts the image plane phase difference distance value D3 Defocus amount and object width measurement value converted from D2 The defocus amount converted from is used to calculate the difference between the data at the same time and the average value of the time-series difference data. If the average value can be calculated with sufficient data, the obtained average value is the image plane phase difference distance value. D3 and object width measurement value D2This represents the difference in absolute error. This average value will be used as the absolute error correction value for data D1508.

[0203] Grounding position distance value D1 When using the grounding position distance value D1 The absolute error is corrected, and if the grounding position distance measurement value D1 was used in S1504, the result will be the same absolute error. In that case, selecting the one with the smaller relative error will reduce the influence of the relative error. In S1509, the distance information generation unit 322 performs an absolute error correction value selection process to decide which result from S1507 or S1508 to select as the absolute error correction value. Details will be described later.

[0204] In S1402, the distance information generation unit 322 generates the distance value measured by imaging plane phase difference distance measurement. D3 The distance information generation unit 322 calculates the distance. D1508 The absolute error correction value is used to correct the amount of defocus. Since the absolute error correction value represents the offset of the defocus, the difference is calculated from the amount of defocus calculated from the input data to obtain the distance measurement value of the D1404. D3 This will be the result after the absolute error has been corrected. The distance value may also be corrected directly. In practice, it will be adjusted to match the absolute error of the data used for the difference in S1507. As mentioned above, the object width measurement value D2 When using the defocus amount converted from, the object width distance value D2 This is the absolute error. Object width measurement value D2 Since this depends on the distance measurement value used in S1504, if the actual object width W was calculated using the ground contact position distance measurement value D1 in S1504, the ground contact position distance measurement value D1 and the imaging plane phase difference distance measurement value D3 , and object width measurement value D2 All of these represent the absolute error of the ground contact position distance measurement value D1. Because the absolute errors are consistent across the three distance measurement values, the integrated distance measurement process in S1406 only needs to consider the relative error when determining the probability distribution. This allows for a simpler and more stable data analysis. D1507 The relative distance value D can be calculated.

[0205] As described above, by inputting the time-series data of the object, the ground contact position distance value D1 and the imaging plane phase difference distance value can be obtained. D3 , and object width measurement value D2 From this, the relative distance value D can be calculated in a time series. From the time series data of the relative distance value D, the relative velocity, relative acceleration, and relative jerk with respect to the target object can be calculated. Using these, the probability distribution of the relative distance value mentioned above can be calculated. For example, the probability distribution can be determined such that the change in relative acceleration is small.

[0206] While we have discussed time-series data so far, it is not always necessary to acquire it continuously. If object recognition fails and correct input data cannot be obtained for that frame, processing can be skipped, and recalculation can be started from that point when correct recognition is achieved. In that case, the average value W' of the actual object width and the absolute error correction value can be used as is, and a stable relative distance value D can be obtained from them. However, in that case, the probability distribution of the relative distance value using relative velocity, relative acceleration, and relative jerk calculated from the relative distance value D cannot be correctly determined, so it is desirable to set it to 0.

[0207] Also, ground contact position distance value D1, image plane phase difference distance value D3 , and object width measurement value D2 It is possible that one or two of these values ​​cannot be calculated correctly. For example, this could occur if the recognized object is floating, or if another object is positioned between the recognized object and the object, preventing accurate measurement of the object's width. In such cases, the change may differ significantly from the previous value or from other distance values, making it highly likely that the value cannot be calculated correctly. In such cases, the influence on the calculation of the relative distance value D can be avoided or mitigated by setting the probability distribution of that distance value to 0 or a very small value.

[0208] S1508 The distance information generation unit 322 then performs a correction process that uses the change in relative distance and the change in the actual object width W to perform a highly accurate absolute error correction when the relative distance value D with respect to the target object has changed sufficiently as a time series change.

[0209] Figure 16 is a schematic diagram illustrating the absolute error correction process. The horizontal axis represents time, with the distance value D plotted on the vertical axis in Figure 16(A) and the actual object width W plotted on the vertical axis in Figure 16(B). Figure 12(B) shows the actual object width W calculated from the object width at each time point, not the average actual object width W'. Relative errors have been removed for illustrative purposes.

[0210] Even when the relative distance value D changes as shown in Figure 16(A), the actual object width W should remain constant as shown by the dotted line in Figure 16(B). However, if absolute error is included, it will change depending on the distance, as shown by the solid line. Using Equation 10, the estimated distance value De with error, using the true distance value Dt, the true actual object width Wt, and the estimated average actual object width We', can be expressed by the following equation (Equation 10).

[0211]

number

[0212] If there is no relative error, then if We and Wt are the same, Dt and De will also be the same, and the relative distance can be estimated correctly. If We is not the same as Wt, that is, if absolute error remains, then the relative distance will also have an error in proportion to the ratio of We and Wt. As a result, the estimated width of the actual object We changes depending on the distance, as shown in Figure 16(B).

[0213] Therefore, when a certain relative distance fluctuation occurs, the absolute distance correction component is determined so that there is no change in the actual object width at each time point. Since the true distance value Dt is unknown, the ground contact position distance measurement value D1 and the image plane phase difference distance measurement value are used. D3 By considering one or more of the relative distance measurement values ​​D, it can be estimated from the variation component and the change in the actual object width W. Specifically, for example, the image plane phase difference distance measurement value D3The actual object width W is calculated using a method equivalent to S1404, and its change in that time series is calculated. The absolute error correction value should be adjusted so that the change in the actual object width W is minimized. A general optimization method can be used to find the correction value that minimizes the change in the actual object width W. This allows for the simultaneous estimation of both the absolute error correction value and the actual object width correction value.

[0214] Also, corrected phase-difference distance values ​​of the imaging plane D3 Object width measurement value D2 The absolute error of the ground contact position distance measurement value D1 should match, and the difference in these absolute errors can also be calculated as the absolute error correction amount. In the ground contact position distance measurement process in S1401, the main cause of the absolute error component is the pitch angle deviation of the imaging device mentioned above. In other words, the vanishing point is at a different position than expected, and this component becomes the absolute error of the ground contact position distance measurement value. In S1508, by estimating the absolute error component, the pitch angle deviation can be estimated. This correction amount can be used as the absolute error correction amount, data D1510. When relative distance fluctuations occur in this way, the absolute error component of each distance measurement value can be corrected with higher accuracy by utilizing this information. If the fluctuations of the absolute error component, including the time series, are within a negligible range, then after the overall absolute error correction process in S1508, the calculation of the average value of the actual object width in the actual object width acquisition process in S1504 and the absolute error correction process in step S1507 are basically unnecessary. Therefore, each process can be simplified, or it can be continued for confirmation.

[0215] In S1509, the distance information generation unit 322 selects which of the absolute error correction values ​​from S1507 or S1508 to output as data D1508. As mentioned above, S1508 is performed when a certain relative distance fluctuation occurs. Therefore, in S1509, the distance information generation unit 322 basically selects the absolute error correction value calculated in S1507. S1508 process is executedIf so, the absolute error correction value calculated in S1508 is selected. If there is a change in the absolute error, the absolute error correction value calculated in S1507 may be selected again. For example, if the absolute error correction value calculated in S1507 changes after the absolute error correction value calculated in S1508 has been selected, the absolute error correction value calculated in S1507 can be selected again. Subsequently, if a certain relative distance change occurs again, S1508 is performed and that absolute error correction value is selected, and in this way, a better absolute error correction value can be continuously selected.

[0216] As described above, Figures 15 and 16 illustrate the flow for correcting relative and absolute errors over time using ground position distance measurement, object width distance measurement, and image plane phase difference distance measurement. Although the above explanation focused on image plane phase difference distance measurement, the same approach can be applied to stereo distance measurement. Relative errors are equivalent to sensor noise effects, while absolute errors can be considered as effects of the installation position of each stereo imaging device. Absolute errors are not converted into defocus values, but the correction amount can be estimated based on the installation position.

[0217] Furthermore, the same approach can be applied to other modalities such as LiDAR. For example, with LiDAR, relative errors in distance resolution occur, and absolute errors also occur as distance offsets, so the same approach as in this embodiment can be applied. In this example, we have explained an integrated flow of three methods, but two methods can be extracted from each, or additional methods such as image plane phase difference ranging, stereo, and LiDAR (other modalities) can be added, and corrections can be applied similarly to four or more methods. Using this method, stable distance values ​​can be obtained for purposes such as tracking a recognized object for a certain period of time. By attaching this imaging device to a vehicle, it can be applied to applications such as ACC (Auto Cruise Control) and autonomous driving.

[0218] <Third Embodiment> The path generation device 150 of the third embodiment achieves high-precision distance measurement from short distances to long distances by combining distance measurement using parallax with multiple images and distance estimation based on a single image.

[0219] Figure 17 is a block diagram showing an example configuration of a distance measuring system to which the present invention is applied. The distance measuring system is assumed to be included in the imaging device 110. The distance measuring system comprises an image sensor 1702, a recognition unit 1721, a distance image generation unit 1712, a scaling distance measuring unit 1703, and a distance correction unit 1704. Here, the image sensor 1702 corresponds to the image sensor 302 of the first embodiment. The recognition unit 1721 corresponds to the recognition unit 321. The distance image generation unit 1712 corresponds to the distance image generation unit 312. The scaling distance measuring unit 1703 and the distance correction unit 1704 correspond to the distance information generation unit 322.

[0220] The image sensor 1702 acquires an image signal from the image sensor 302. The image sensor 1702 acquires an image signal composed of the image signals acquired by the first photoelectric conversion unit 411 and an image signal composed of the image signals acquired by the second photoelectric conversion unit 412. These are images corresponding to different viewpoints and are called disparity images. The image sensor 1702 also acquires a composite image signal obtained by combining the image signals of the two disparity images as the captured image. The image sensor 1702 may acquire one of the two disparity images as the captured image.

[0221] Furthermore, the camera configuration for obtaining the parallax image may use stereo cameras arranged side by side. Alternatively, a monocular camera configuration may be used, and the parallax image may be obtained by considering the speed of the vehicle and treating the relative movement of objects in consecutive frame images as parallax.

[0222] The recognition unit 1721 applies image recognition processing to the image captured by the image sensor 1702 to detect objects contained in the image. In order to realize automatic driving control and collision mitigation braking control of the vehicle 100, it is necessary to recognize objects such as the lane in which the vehicle 100 is traveling, vehicles traveling in front of the vehicle 100 (preceding vehicles), and people on the sidewalk. As object detection methods, template matching methods that detect objects with a nearly constant appearance (such as traffic signals and traffic signs) and methods that use machine learning to detect general objects (such as vehicles and people) are widely known.

[0223] In this embodiment, the recognition unit 1721 performs a lane marking detection task and an object recognition task. Figure 18 is a schematic diagram showing the output results of the lane marking detection task and the object recognition task performed by the recognition unit 1721. The lane marking detection task takes the captured image as input and uses a machine learning model to detect whether each pixel is a lane marking (or white line) on the road, thereby obtaining a lane marking area map labeled as such (in the figure, detected lane marks are shown as black dashed lines). The object recognition task takes the captured image as input and uses a machine learning model to detect objects on the road, determining the type of detected object (person, car, sign), the coordinates of the upper left point (x0, y0) and the lower right point (x1, y1) of the detection frame, and obtaining the coordinates of the detection frame touching the detected object. Here, the output of the object recognition task is assumed to be equivalent to the external information shown in the first embodiment.

[0224] The distance image generation unit 1712 obtains distance data from the disparity image obtained by the image sensor 1702. Distance measurement from the disparity image can be performed by detecting corresponding points between images with different viewpoints to calculate the disparity value, and then calculating the distance from that disparity value and the camera conditions (focal length, baseline length) used to capture the disparity image. As described above, even if the camera used to calculate the disparity image is a monocular camera using a dual-pixel CMOS sensor, it is possible to identify the camera conditions for distance calculation. Generally, it is known that in distance measurement using disparity images, the accuracy of distance estimation deteriorates because the disparity almost disappears when the object to be measured is far away.

[0225] The scaling distance measurement unit 1703 calculates the distance value of the second region by scaling the distance value of the first region calculated by the distance image generation unit 1712 according to the size ratio of objects in the first region and objects in the second region. In this embodiment, an example is described in which the distance value on the near side calculated by the distance image generation unit 1712 is scaled by extending the road surface on the near side to the far side and using the road information on both the near and far sides.

[0226] Figure 19 illustrates the relationship between the camera mounting position and the road surface. A global coordinate system (X,Y,Z) is set with the road surface, which is vertically aligned with the camera mounting position V(0,Y0,0), as the origin O. The imaging camera used for the image sensor 1702 is assumed to be positioned horizontally along its optical axis. The field of view of the imaging camera indicates the area to be captured. If the image width of the imaging area, i.e., the image width of the captured image and the distance map calculated by the distance image generation unit 1712, is wu and the image height is hv, then the center pixel of the distance map is expressed as c(wu / 2,hv / 2).

[0227] First, the road surface on the near side is estimated. The area assumed to be the near-side road surface on the distance map is the lower part of the distance map. Alternatively, it is also possible to directly estimate the near-side road surface by recognition processing of the captured image. Within the lower part of the distance map, the area around pixels determined to be lane lines on the lane line area map detected by the recognition unit 1721 may be considered the road surface. Only the distance map area determined to be closer than a threshold determined by camera installation conditions (field of view, resolution, line of sight angle), shooting environment (weather, time of day), and information about the road on which the vehicle is traveling (lane width, number of lanes, branching / merging points, road type) may be considered the road surface.

[0228] If we let D be the distance value of pixel p(u0,v0) on the distance map, then the nearby road surface p(u0,v0) can be transformed into global coordinates X=u0-wu / 2, Y=v0-hv / 2, Z=D.

[0229] Assuming the road surface is horizontal, the equation of the surface is expressed as aX + bY + cZ + d = 0 (where a, b, c, and d are constants). The road surface can be estimated by determining the constants in the above equation using four or more points that represent the road surface on the near side.

[0230] The estimated road surface equation makes it possible to estimate the road surface on the far side. Furthermore, the distance is scaled from this extended far-side road surface. Specifically, the depth Z of point R(X,Y,Z) located at the intersection of the road surface equation and a line passing through the viewpoint V(0,Y0,0) and pixel q(u1,v1) which represents the far-side region on the distance map can be estimated as a distance value.

[0231] Even if the scaling distance measurement unit 1703 cannot obtain the road distance corresponding to pixel q(u1,v1) from the distance data, it is possible to obtain the road distance value Z corresponding to q(u1,v1) by performing the calculation as described above.

[0232] If we consider this process on a distance map, the distance in the first region is scaled by the ratio of road surfaces in the near-field region (first region) to road surfaces in the far-field region (the depth ratio in 3D space) to calculate the distance in the second region.

[0233] The distance values ​​calculated in this way do not use the distance values ​​from the parallax distance measuring unit, which deteriorates at long distances, thus improving the accuracy of distance estimation at long distances.

[0234] Furthermore, the distance measurement accuracy can be improved by using the detection frame detected by the recognition unit 1721. Figure 20 is a schematic diagram showing a scene in which two signs of known size are detected on a road. Assume that in the acquired image, one sign of known size is placed on the near side and one on the far side. In this state, the distance to the near-field sign can be accurately calculated by the distance image generation unit 1712. Also, if the object size is known, the scaling distance measurement unit 1703 can calculate the distance to the far-field sign based on the ratio of the number of pixels in the image.

[0235] Let w0 be the height (number of pixels) of the sign at the near distance, d0 be the distance from the camera position, and w1 be the height (number of pixels) to the sign at the far distance, and d1 be the distance from the camera position. The scaling distance measuring unit 1703 can calculate the distance using the formula d1 = d0 * (w0 / w1). As described above, if objects of known size (such as signs and traffic signals) or objects that can be assumed to have the same size at the near and far distances (such as guardrails, road markings, width, length, and spacing) can be detected, high-precision scaling distance measurement can be performed. When performing scaling distance measurement with objects of known size, it is difficult to improve the accuracy of the entire distance range from near to far. However, by combining the scaling distance measurement that extends the road surface as described above with the scaling distance measurement that utilizes the object size ratio, it becomes possible to perform high-precision scaling distance measurement.

[0236] The distance correction unit 1704 corrects the distance value measured by the distance image generation unit 1712 based on the distance value calculated by the scaling distance measurement unit 1703 to obtain a corrected distance value Dc. Hereinafter, the value measured by the distance image generation unit 1712 will be called the distance value D, and the distance value calculated by the scaling distance measurement unit 1703 will be called the scaling distance measurement value Ds. The corrected distance value Dc is calculated according to the following equation (Equation 11), with coefficient α.

[0237]

number

[0238] The coefficient α is determined by one of the following three methods.

[0239] (1) Method for determining the coefficient α based on the magnitude of the distance value Regarding the distance measurement by the parallax distance measuring unit, the distance value to the object being measured itself affects the measurement accuracy. As the distance to the object increases, the distance accuracy of the distance value calculated by the parallax distance measuring unit decreases. Therefore, the coefficient α is determined so that the proportion of the scaled distance measurement value increases according to the distance value. This allows for obtaining accurate measurement results. In other words, the coefficient α is determined so that the larger the distance value D, the smaller α becomes.

[0240] (2) Method for determining the coefficient α based on the contrast of the object to be measured Regarding the distance measurement of the parallax distance measuring unit, in addition to the distance value to the object being measured, the contrast of the object is another factor that affects the measurement accuracy. When calculating the corresponding points (parallax) between parallax images, even if low-contrast areas are matched, the distinction from the surrounding areas is unclear, and the parallax cannot be accurately determined. Therefore, if the object being measured is not sufficiently illuminated at night and its contrast is low, the measurement accuracy of the distance value is judged to be low, and the coefficient α is set so that the proportion of the scaled distance value is high. In other words, the coefficient α is determined so that it is large when the contrast of the object being measured is low. Since the scaled distance value is generated based on the highly accurate distance value of the area illuminated by, for example, vehicle lighting, accurate measurement results can be obtained even when the contrast of the object being measured is low.

[0241] (3) Method for determining coefficients based on the category type of the detection frame Depending on the type (category) of the distance measurement target, the distance measurement accuracy tends to be different for parallax distance measurement and scaling distance measurement respectively. For example, for a distance measurement object that is far from the scaling reference such as the light-emitting part of a traffic signal, parallax distance measurement can measure the distance without distinguishing it from other objects, but the distance measurement accuracy tends to decrease in scaling distance measurement. Therefore, when the category of the detection frame is a specific type, by setting the coefficient so that the ratio of the distance value is increased, a distance measurement result with high accuracy can be obtained. Also, as shown in FIG. 20, when the distance accuracy of scaling distance measurement in the peripheral area where an object is detected increases by the method of scaling distance measurement, the coefficient may be determined so that the ratio of scaling distance measurement around the detection frame increases according to this category.

[0242] The coefficient determination method shown above does not need to be limited to one, and the final coefficient may be determined based on the coefficients generated for each factor. By the above, a distance measurement result with high accuracy can be obtained.

[0243] (Modified example of the scaling distance measurement unit) As a modified example of the scaling distance measurement unit 1703, an example of estimating the distance to the target object by scaling the distance data in the vicinity based on the size ratio of the road width on the images on the near side and the far side will be described. FIG. 21 is a block diagram showing a configuration example of the scaling distance measurement unit 170३.

[0244] The scaling distance measurement unit 1703 includes a lane analysis unit 2101, a roll angle estimation unit 2102, a ground contact position estimation unit 2103, and an object distance calculation unit 2104.

[0245] The lane analysis unit 2101 detects the number of pixels between the lane lines as the lane width from the lane line area map, and detects the central coordinates between the lane lines as the lane center (coordinates).

[0246] FIG. 22 is a schematic diagram schematically showing the processing of lane width detection and lane center detection executed by the lane analysis unit 2101. FIG. 22(A) is a diagram schematically showing a lane line area map indicating lane lines in an image. Pixels detected as lane lines in the image by the lane analysis unit 2101 are given (represented by black dashed lines in the figure) lane line flags. Since lane lines on the road are shown as dashed lines at the boundary between the driving lane and the overtaking lane, or the lane lines are hidden by vehicles or obstacles on the road, the lane line area map appears intermittently.

[0247] FIG. 22(B) is a schematic diagram showing a method for calculating the lane width and the position of the center of the lane. The lane analysis unit 2101 sequentially checks, pixel by pixel from the left side to the right side of the lane line area map, whether the lane line flag is set. If the pixel being inspected does not have a lane line flag and the pixel to the left adjacent of the pixel being inspected has a lane line flag, the pixel being inspected is taken as the starting point of the road width. If the pixel being inspected has a lane line flag and the pixel to the left adjacent of the pixel being inspected does not have a lane line flag, the pixel being inspected is taken as the end point of the road width. Thereby, as shown by the arrow group in FIG. (B), for each line of the lane line area map, the lane width (the length of the arrow) and the position of the lane (the central position of the arrow) can be detected one by one.

[0248] FIG. 22(C) is a schematic diagram showing the detected lane width data. The lane width is observed as the number of pixels proportional to the reciprocal of the distance from the camera as long as the actual size of the lane does not change. In FIG. 22(C), the horizontal axis represents the line of the lane line area map, and the vertical axis represents the detected lane width (number of pixels). When shown in this way, the lane width shows a highly linear relationship with respect to the line. Due to the intermittency of the lane lines, etc., the lane width of one lane and the lane width of two lanes may be observed mixedly, but since there is a difference of approximately two times, it is easy to separate them.

[0249] Figure 22(D) shows a schematic diagram illustrating the interpolation of observed data for the separated left lane and data for lines where lane width could not be obtained from the observed data. Regarding the separation of adjacent lanes (left and right lanes), it is easy to exclude observed data for adjacent lanes by comparing the position of the lanes (center position of the arrows). From this information, the lane width for only the left lane shown in Figure 22(B) can be obtained using robust estimation methods such as the RANSAC method. For lane width interpolation, the equation of an approximate straight line can be calculated using the RANSAC method described above, and this approximate straight line equation can be used, or interpolation can be performed by interpolating between the observed road width data. Regarding the position of the lanes, data for each line can be obtained using the same method as described above.

[0250] Using the lane information obtained above (lane width and lane center), scaling distance measurement is performed.

[0251] The roll angle estimation unit 2102 estimates the camera's roll angle from the distance map. Several factors can cause a roll angle for an in-vehicle camera. For example, uneven road surfaces can cause a difference in height between the contact points of the left and right tires, preventing the camera mounting position from remaining horizontal and resulting in a roll angle. Also, centrifugal force when a vehicle turns a curve can deform the vehicle body itself, generating a roll angle. Such roll angles significantly affect distance estimation.

[0252] The scaling distance measurement described here is a process that scales the distance to an object on the road surface using the ratio of the number of lane width pixels assumed to be at the same distance as the object to the number of neighboring lane width pixels, thereby scaling the distance to the neighboring object with high accuracy. When the camera rolls, it becomes difficult to find the lane width assumed to be at the same distance as the object.

[0253] Figure 23 is a schematic diagram comparing the position of lane width in captured images with and without roll angle. The image on the left shows the case without roll angle, while the image on the right shows the same scene with a roll angle of 10° centered on the lower left of the image. In the image on the left, the ground contact position of the vehicle being measured and the lane assumed to be at the same distance are aligned on the same straight line, making correspondence easy. On the other hand, in the image on the right where a roll angle is present, there is no lane on the same line as the ground contact position of the vehicle being measured, and even if there were a lane, it would be impossible to calculate the lane width at the same distance. To find the lane width at the same distance as the vehicle being measured, it is necessary to estimate the roll angle accurately.

[0254] The roll angle estimation unit 2102 estimates the roll angle based on the distance map obtained by parallax distance measurement. When a roll angle occurs, it is known that the position at which the distance to the road surface, which can be assumed to be horizontal, is the same, is tilted, as shown in Figure 23. Since the roll angle is a rotation around the viewpoint direction, there is no change in the distance between the camera and the target object, and the image captured by the camera is tilted due to the roll angle, so the distance map also tilts in line with the tilt of the road surface in the image.

[0255] Figure 24 is a flowchart showing the roll angle estimation process performed by the roll angle estimation unit 2102. Figure 25 is a schematic diagram illustrating each step in the roll angle estimation process.

[0256] In S2400, the roll angle estimation unit 2102 determines the target pixel setting range from the distance map. Figure 25(A) is a schematic diagram showing the target pixel setting range in the distance map.

[0257] As shown in Figure 25(A), the area to be set to the pixel of interest is located in the lower left region of the distance map. The height of the area to be set to the pixel of interest is assumed to correspond to the road surface, for example, from the horizontal line to the bottom edge of the distance map. The width of the area to be set to the pixel of interest is set taking into account the distance between the pixel of interest and the search area. The search area is set at a position a predetermined number of pixels (a predetermined interval) away horizontally from the pixel of interest. If the distance between the pixel of interest and the search area is large, even if the width of the area to be set to the pixel of interest is set to be large, it will not be possible to set the search area. In this embodiment, the predetermined interval is set to about 1 / 4 of the image width of the distance map, and the size of the area to be set to about half of the image width of the distance map. By appropriately setting the area to be interest, the amount of computation can be reduced.

[0258] In S2401, the roll angle estimation unit 2102 obtains distance data for the target pixel 2500 from the distance map. In this embodiment, distance data is obtained for the target pixel 2500 within the target pixel setting range.

[0259] In S2402, the roll angle estimation unit 2102 determines the search range. Regardless of whether a roll angle occurs or not, the distance values ​​of pixels near the pixel of interest tend to be close to the distance value of the pixel of interest. Therefore, in order to detect the roll angle with high resolution from the coordinates of the pixel of interest 2500 and the corresponding pixel 2501, the roll angle estimation unit 2102 sets a search range that is spaced horizontally at a predetermined interval from the pixel of interest 2500.

[0260] The roll angle estimation unit 2102 can limit the height of the search range by the expected roll angle range. For example, in the case of an in-vehicle camera installed in a vehicle, when driving on a typical road, the roll angle is limited to ± a few degrees. In this embodiment, it is set to about 1 / 8 of the image height of the distance map. The width of the search range is set from a position a predetermined distance horizontally from the target pixel 2500 to the right edge of the distance map. The right edge of the search range is not limited to the right edge of the distance map, but if the width of the search range is small, the area may be too small to find the distance data of the road surface corresponding to the target pixel 2500, so it is preferable to set the width of the search range as large as possible.

[0261] In S2403, the roll angle estimation unit 2102 searches for a corresponding pixel 2501 that corresponds to the target pixel 2500 within the search range. To position a pixel similar in distance value to the target pixel 2500 within the search range as the corresponding pixel 2501, the difference between the distance value of the target pixel 2500 and each pixel within the search range is detected, and the pixel with the smallest difference is designated as the corresponding pixel 2501. Note that the method for searching for the corresponding pixel 2501 is not limited to comparing the difference between one pixel and another; a difference comparison of distance values ​​between neighboring pixel groups including the target pixel and pixels within the search range may be performed, and the central pixel of the pixel group with the highest similarity may be designated as the corresponding pixel 2501.

[0262] In S2404, the roll angle estimation unit 2102 calculates the inclination θ from the coordinates of the pixel of interest and the corresponding pixel. As shown in Figure 25(B), if the coordinates of the pixel of interest 3800 are (x0, y0) and the coordinates of the corresponding pixel 2501 are (x1, y1), the inclination θ is calculated by θ = arctan( (y1-y0) / (x1-x0) ).

[0263] In S2405, the roll angle estimation unit 2102 branches depending on whether it has completed processing all pixels within the target pixel setting range. If it has completed processing all pixels within the target pixel setting range as target pixels, it proceeds to S2406. If it has not completed processing all pixels within the target pixel setting range as target pixels, it proceeds to S2401 and processes new target pixels.

[0264] In S2406, the roll angle estimation unit 2102 calculates the roll angle. FIG. 25(C) is a schematic diagram showing the horizontal axis as the distance value of the target pixel and the vertical axis as the slope calculated for the target pixel. Since the slope calculated from one target pixel includes noise components, a plausible roll angle is detected by averaging a plurality of slopes. The distance map referred to for roll angle estimation is generated from the disparity image, but it is known that the detection accuracy of distance measurement from the disparity image deteriorates as the distance to the object increases. Therefore, when calculating the slope for each target pixel using the distance map, the variation in the calculated slopes increases according to the magnitude of the distance value of the target pixel. Therefore, in calculating the roll angle, the roll angle is estimated by weighted averaging such that the ratio of the slopes with a small distance value of the target pixel is large and the ratio of the slopes with a large distance value is small. Incidentally, the similarity in the corresponding pixel search may be used as a factor for determining the ratio of the weighted average.

[0265] By the above processing, the roll angle can be estimated using the distance map of the road surface.

[0266] Furthermore, a method for setting the interval between the target pixel and the search range when the resolution of the roll angle to be estimated is predetermined will be described.

[0267] The interval between the target pixel and the search range is determined by the resolution of the roll angle. The roll angle is calculated from the slope from the target pixel to the corresponding pixel, and is represented by the ratio of the horizontal difference and the vertical difference between the target pixel and the corresponding pixel. Since the vertical difference is at least one pixel, the resolution of the roll angle is determined by the magnitude of the horizontal difference. The interval d between the target pixel and the search range is obtained using the following formula (Equation 12) using the resolution r of the roll angle.

[0268]

Equation

[0269] Equation 12 shows the minimum detection angle, which is derived from the relationship between the angle r for a difference of 1 pixel in the y-axis direction, where d is the distance between the pixel of interest and the search range.

[0270] The larger the calculated interval d, the higher the resolution. However, as mentioned earlier, increasing the interval d narrows the search range, so it is preferable to set a minimum resolution. For example, if the required resolution r is 0.1°, the interval d will be 573 pixels or more.

[0271] The above calculation method allows for setting an appropriate interval when the roll angle detection resolution is given.

[0272] The ground contact position estimation unit 2103 uses the detection frame obtained by the object detection task, the lane center determined by the lane analysis unit 2101, and the roll angle estimated by the roll angle estimation unit 2102 to estimate the coordinates of the lane width data located at the same distance as the ground contact position of the distance measurement target. Figure 26 is a flowchart showing the coordinate estimation process of the lane width data performed by the ground contact position estimation unit 2103.

[0273] In S2601, the ground contact position estimation unit 2103 selects a detection frame for distance measurement from the detection frames obtained in the object recognition task.

[0274] In S2602, the grounding position estimation unit 2103 sets the center position of the lower part of the detection frame as the coordinates (xc,yc) of the grounding position of the distance measurement target. Figure 27 is a schematic diagram showing the coordinates (xc,yc) of the grounding position of the distance measurement target set by the grounding position estimation unit 2103. The coordinates (xc,yc) of the grounding position of the distance measurement target are expressed using the upper left coordinate (x0,y0) and lower right coordinate (x1,y1) of the detection frame, using the following equation (Equation 13).

[0275]

number

[0276] Furthermore, a method other than the one described above may be used to obtain the coordinates of the ground contact position of the object to be measured. With wide objects such as cars, the object will appear tilted by the roll angle, which can lead to a large discrepancy between the center coordinate of the lower edge of the detection frame and the coordinates where the tire is actually in contact with the ground. In such cases, for example, the contour of the object within the detection frame may be detected using the captured image or a distance map, and the position where a straight line R with a roll angle inclination touches the lower edge of the object's contour may be set as the coordinates of the ground contact surface. By using such a configuration, it is possible to set the coordinates where the object to be measured is actually in contact with the ground.

[0277] Since the process of recognizing the contour of an object is computationally intensive, a configuration that takes computational cost into consideration may be used to select an appropriate coordinate from the corners of the detection frame depending on the positional relationship between the lane center and the object to be measured. Specifically, one selection method is to select the coordinate of the lower left of the detection frame if the object to be measured is to the left of the lane center, and the coordinate of the lower right of the detection frame if it is to the right. This may allow the straight line R, which will be determined in the next step, to be set to a coordinate where the object is actually in contact with the ground, which is closer than setting the center of the lower edge of the detection frame.

[0278] In S2603, the ground contact position estimation unit 2103 determines the coordinates (xt, yt) where a straight line R passing through the ground contact position coordinates (xc, yc) of the distance measurement target intersects with the straight line C of the lane center obtained in the lane mark detection task.

[0279] By processing as described above, even if the distance map is tilted due to the roll angle, it is possible to determine the coordinates of the lane width data that are at the same distance from the ground contact point of the distance measurement target.

[0280] The object distance calculation unit 2104 calculates the distance to the object to be measured using the coordinates on the lane width data corresponding to the ground contact position obtained by the ground contact position estimation unit 2103, the lane width data calculated by the lane analysis unit 2101, and the data on the distance map. Figure 28 is a flowchart showing the distance estimation process to the object to be measured performed by the object distance calculation unit 2104.

[0281] In S2801, the object distance calculation unit 2104 calculates the lane width N2 using the lane width data and coordinates (xt, yt). The object distance calculation unit 2104 may obtain the lane width at coordinates (xt, yt) as the lane width N2, or it may obtain multiple lane widths near coordinates (xt, yt), weight them according to the distance from coordinates (xt, yt), and obtain the weighted average. When determining the lane width from lanes detected by image recognition, there is a possibility of discrepancies between the detected lane width and the actual lane width due to errors in image recognition. As described above, by calculating the lane width N2 using multiple lane widths, the discrepancy in the detected lane width can be reduced. Depending on the processing load, the number of lane width data used for smoothing may be limited to one.

[0282] In S2802, the object distance calculation unit 2104 associates and saves the lane width data and distance data obtained in the current frame. First, the object distance calculation unit 2104 obtains each lane width data and its coordinates, and obtains distance data corresponding to the coordinates of the lane width data from the distance map. At this time, the corresponding distance data may be obtained by obtaining distance data corresponding to each coordinate within the range of the lane width data (xn-N1 / 2, yn) to (xn+N1 / 2, yn) centered on the coordinate (xn, yn) corresponding to the lane width N1 of the lane centerline C, and then smoothing it. The object distance calculation unit 2104 performs this association of lane width data and distance data for each lane width data obtained in the current frame T0. Figure 29 is a schematic diagram showing each lane width data. The object distance calculation unit 2104 may narrow the range of data for which it performs the association depending on the processing load and recording capacity, for example, it may be configured to save only the data for locations where the accuracy of the distance data is high.

[0283] In S2803, the object distance calculation unit 2104 calculates reference data B to be used to calculate the distance to the distance to be measured. When there are k data points associated in the previous step, the reference data B is expressed by the following equation (Equation 14) using the distance to the lane D1 and the lane width N1.

[0284]

number

[0285] At this time, since the error in distance data tends to increase as the distance increases, the reference data B may be calculated using a weighted average in which the weight corresponding to the small distance value is increased and the weight corresponding to the large distance value is decreased. Alternatively, the data obtained in S2802 may be stored as past frame data for several frames, such as T1 and T2, and this may be included in the calculation.

[0286] In equation 13, D1[n]×N1[n] is the value corresponding to the lane width used as the basis for distance calculation. If D is the distance to the lane, N is the number of pixels representing the lane width in the image, F is the horizontal field of view of the imaging camera, and H is the horizontal image width, then the actual lane width W can be calculated by W=(D×N)×2×tan(F / 2) / W. The 2×tan(F / 2) / W part can be treated as a fixed value once the specifications of the imaging camera are determined. Therefore, the actual lane width becomes the variable (D×N). Due to this relationship, in equation 13, since the lane width can be considered the same when a vehicle is driving, it is possible to suppress observation noise by applying smoothing to D1[n]×N1[n]. Furthermore, even in multiple consecutive frames, since it is assumed that the vehicle is driving in the same lane, the actual lane width can be considered the same, and it is possible to suppress observation noise by applying smoothing to D1[n]×N1[n].

[0287] On the other hand, in some frames, the image may be blurred due to the vehicle running over small stones, potentially preventing the correct acquisition of lane width data. Therefore, it may be possible to reduce the error from the reference data by reducing the weight of frames with a large deviation in D1[n]×N1[n] (i.e., large blur). Alternatively, the weight may be increased as the time difference from the current frame decreases, and decreased as the time difference increases, in order to balance the response to the current frame with the smoothing of blur that occurred between frames.

[0288] In S2804, the object distance calculation unit 2104 uses the reference data B obtained in the previous step and the lane width N2 of the distance to be measured to determine the distance data D2 for the distance measurement. The distance data D2 is determined by the following equation (Equation 15).

[0289]

number

[0290] By processing as described above, the distance to the object to be measured can be estimated. The object distance calculation unit 2104 outputs object distance information indicating the distance to the object to be measured, which has been determined in this way.

[0291] As described above, this embodiment explains an example of a distance measuring system that can measure distances with high accuracy from near to far by combining stereo distance measurement using parallax images and distance estimation from a single image. Even at long distances where distance measurement accuracy deteriorates with distance measurement using parallax images alone, distance accuracy can be maintained even when the object to be measured is far away by using the size ratio of a scaling object such as a road surface. When the object to be scaled is an object whose actual size does not change easily, such as a road, distance accuracy can be improved by reducing observation noise during size measurement through smoothing during scaling. Furthermore, by changing the composite ratio according to the distance range in distance measurement using parallax images and distance estimation using a single image, more robust distance estimation becomes possible. In addition, by detecting and correcting the roll angle, robust distance estimation that reduces the effects of vehicle deformation and unevenness of the road surface becomes possible.

[0292] <Other Embodiments> In the embodiments described above, an imaging device was explained as an example in which left and right parallax images are acquired using a phase-difference imaging method via the same optical system to acquire distance image data. However, the method of acquiring parallax images is not limited to this. It is also possible to acquire left and right parallax images using a so-called stereo camera, in which a left parallax image and a right parallax image are acquired by two imaging devices placed at a predetermined distance apart.

[0293] Furthermore, it is also possible to perform the aforementioned distance measurement using distance measurement devices such as LiDAR to acquire distance information and external information obtained through image recognition of the captured images obtained from the imaging device.

[0294] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of symbols]

[0295] 100 vehicles 101 Driver 110 Imaging device 120 Radar equipment 130 Route Generation ECU 140 Vehicle Control ECU 150 Route Generator 160 Vehicle Information Measuring Instrument

Claims

1. Image acquisition means for acquiring captured images and depth images, A detection means for detecting an object from the aforementioned IM image, A first distance information acquisition means that acquires first distance information indicating the distance to an object detected by the detection means from the distance image, A ground contact position acquisition means that acquires the ground contact position of an object detected by the detection means based on the captured image, A third distance information acquisition means acquires third distance information indicating the distance to the object detected by the detection means, based on information about the size of the object in the image signal generated by the imaging means. A second distance information acquisition means selects one of the first distance information, the grounding position acquired by the grounding position acquisition means, and the third distance information, and acquires it as second distance information indicating the distance to the object detected by the detection means. An information processing device characterized by having the following features.

2. The information processing apparatus according to claim 1, characterized in that the ground contact position of the object is the lower end of the region of the object detected by the detection means.

3. The information processing apparatus according to claim 1, characterized in that the captured image and the distance image are generated from the image signal.

4. The information processing apparatus according to claim 3, characterized in that the distance image is generated from two image signals generated by the imaging means.

5. The imaging means generates a first image signal generated by a first photoelectric conversion unit of the image sensor in the imaging means and a second image signal generated by a second photoelectric conversion unit of the image sensor in the imaging means. The distance image is generated based on the relative positional shift between the first image signal and the second image signal. The information processing apparatus according to claim 3.

6. The imaging means is composed of a plurality of imaging devices, The aforementioned distance image is generated from two image signals generated by different imaging devices among the plurality of imaging devices. The information processing apparatus according to claim 3.

7. The information processing apparatus according to claim 1, characterized in that the second distance information acquisition means acquires the second distance information by integrating the first distance information and the grounding position.

8. The information processing apparatus according to claim 1, characterized in that the second distance information acquisition means acquires the second distance information, which is either the first distance information or the distance information indicating the distance to the grounding position.

9. The information processing apparatus according to claim 1, characterized in that the second distance information acquisition means acquires the second distance information by integrating the first distance information, distance information indicating the distance to the grounding position, and the third distance information.

10. The third distance information acquisition means is characterized by acquiring the size of the object using at least one of the distance information, the first distance information and the distance information indicating the distance to the grounding position. The information processing apparatus according to feature 1.

11. The information processing apparatus according to claim 1, further comprising a correction means for correcting the first distance information and distance information indicating the distance to the grounding position based on the change in the size of the object used in the third distance information acquisition means.

12. Image acquisition means for acquiring captured images, A detection means for detecting an object from the aforementioned IM image, A first distance information acquisition means that acquires first distance information indicating the distance to an object detected by the detection means from distance information acquired from a distance measuring device, A ground contact position acquisition means that acquires the ground contact position of an object detected by the detection means based on the captured image, A third distance information acquisition means that acquires third distance information indicating the distance to the object detected by the detection means, based on information about the size of the object in the image signal generated by the imaging means, A second distance information acquisition means selects one of the first distance information, the grounding position acquired by the grounding position acquisition means, and the third distance information, and acquires it as second distance information indicating the distance to the object detected by the detection means. An information processing device characterized by having the following features.

13. The information processing apparatus according to claim 1 or 12, characterized in that the second distance information acquisition means acquires the distance information with the highest probability of existence of the object from among the first distance information and the distance information indicating the distance to the grounding position as the second distance information.

14. The information processing apparatus according to claim 13, characterized in that the second distance information acquisition means calculates the distribution of the probability of existence of the current distance value using any of the distance information, velocity information, acceleration information, or jerk information in a time series.

15. Image acquisition step to obtain an image capture image and a depth image, A detection step of detecting an object from the captured image, A first distance information acquisition step is to acquire first distance information indicating the distance to the object detected by the above detection step from the distance image, A grounding position acquisition step, which acquires the grounding position of the object detected by the detection step based on the captured image, A third distance information acquisition step is performed to acquire third distance information indicating the distance to the object detected by the detection step, based on information about the size of the object in the image signal generated by the imaging step. An acquisition step which selects one of the first distance information, the grounding position acquired by the grounding position acquisition step, and the third distance information and acquires it as the second distance information, A control method for an information processing device, characterized by having the following features.

16. Image acquisition step to acquire captured images, A detection step of detecting an object from the captured image, A first distance information acquisition step is to acquire first distance information indicating the distance to the object detected by the above detection step from distance information acquired from a distance measuring device, A grounding position acquisition step, which acquires the grounding position of the object detected by the detection step based on the captured image, A third distance information acquisition step is performed to acquire third distance information indicating the distance to the object detected by the detection step, based on information about the size of the object in the image signal generated by the imaging step. A second distance information acquisition step involves selecting one of the first distance information, the grounding position acquired by the grounding position acquisition step, and the third distance information, and acquiring it as the second distance information. An information processing device characterized by having the following features.

17. A computer-readable program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 14.

Images