Electronic devices, mobile devices, distance calculation methods, and computer programs

Abstract

To provide an electronic device, etc. that are able to acquire a distance to an object highly accurately based on information on the object included in an image.SOLUTION: An electronic device includes: first distance information acquiring means configured to acquire first distance information corresponding to an object included in an image signal; second distance information acquiring means configured to acquire second distance information, based on at least one of information on an end position of the object included in the image signal and information on a size of the object included in the image signal; distance information integrating means configured to combine the first distance information and the second distance information to generate integrated distance information; thereby acquiring a distance to the object highly accurately based on the information on the object included in the image.SELECTED DRAWING: Figure 14

Description

Technical Field

[0005]

[0001] The present invention relates to an electronic device for acquiring the distance from a moving object to an object, a moving object, a distance calculation method, a computer program, and the like.

Background Art

[0002] There is an imaging device having a sensor in which a plurality of pixel regions having a photoelectric conversion function are two-dimensionally arranged, and in each pixel region, an image signal and distance information can be acquired. The solid-state imaging device described in Patent Document 1 arranges pixels having a distance measuring function in some or all of the pixels of the imaging device, and detects the subject distance based on the phase difference detected on the imaging surface (imaging surface phase difference method).

[0003] That is, a displacement is calculated based on the correlation relationship between two image signals based on images generated by light beams passing through different pupil regions of the imaging optical system included in the imaging device, and the distance is acquired based on the displacement. The correlation relationship between the two image signals is evaluated using a method such as a region-based matching method in which image signals included in a predetermined collation region are cut out from each image signal and the correlation relationship is evaluated.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when there is little change in the contrast of the subject included in the image signal, or when the amount of noise included in the image signal is large, etc., misevaluation of the correlation relationship may occur due to the subject or shooting conditions. When the occurrence of misevaluation of the correlation relationship exceeds a certain level, the amount of positional deviation between the two calculated image signals has an error, and the accuracy of the acquired distance may decrease.

[0006] The present invention aims to provide electronic devices, etc., that can accurately determine the distance to an object based on information about the object contained in an image. [Means for solving the problem]

[0007] An electronic device relating to one aspect of the present invention is, Corresponding to objects contained in the image signal Based on the region, using a phase difference distance measurement method A first distance information acquisition means for acquiring first distance information, The object included in the image signal lower end Location information Based on the positional relationship between the vanishing point information and A second distance information acquisition means for acquiring second distance information, The object included in the image signal Width or height Based on the information There, A third distance information acquisition means for acquiring third distance information, With the aforementioned object Includes relative velocity, relative acceleration, and relative jerk parameters. Motion models estimated from velocity information This is the distance measurement value estimated by A fourth distance information acquisition means for acquiring a fourth distance information, The first distance information, the second distance information, the third distance information, and the fourth distance information By using weight coefficients corresponding to each of the above, the first distance information, the second distance information, the third distance information, and the fourth distance information are weighted and integrated. It includes distance information integration means for generating integrated distance information. [Effects of the Invention]

[0008] The present invention makes it possible to realize electronic devices that can acquire the distance to an object with high precision based on information about the object contained in an image. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram showing an example of the vehicle configuration according to the first embodiment. [Figure 2] This is a block diagram showing an example of the vehicle configuration according to the first embodiment. [Figure 3] This is a functional block diagram showing an example configuration of a route generation device according to the first embodiment. [Figure 4] (A) and (B) are schematic diagrams showing a configuration example of an image pickup device according to the first embodiment. [Figure 5] (A) to (D) are schematic diagrams for explaining the relationship between the subject distance and incident light in the imaging surface phase difference method. [Figure 6] (A) and (B) are flowcharts showing examples of processes executed by the image processing unit according to the first embodiment. [Figure 7] (A) and (B) are flowcharts showing examples of processes executed by the distance information generation unit according to the first embodiment. [Figure 8] (A) to (E) are schematic diagrams showing examples of images and information in an example of a process performed by the distance information generation unit according to the first embodiment. [Figure 9] It is a flowchart showing an example of a path generation process executed by the path generation unit according to the first embodiment. [Figure 10] It is a flowchart showing an example of an object distance information generation process performed by the distance information generation unit in Modification 2. [Figure 11] It is a schematic diagram for explaining an example of the temporal change of the object distance information of an object with the same identification number as the Nth object. [Figure 12] It is a flowchart showing an example of an object distance information generation process performed by the distance information generation unit in Modification 3. [Figure 13] (A) to (D) are schematic diagrams for explaining weighted averaging. [Figure 14] It is a flowchart showing an example of a process for acquiring a distance measurement value (object distance information) in the second embodiment. [Figure 15] It is a flowchart showing a part of an example of a process when a plurality of input data continuous in time series in the second embodiment are input as one according to the third embodiment. [Figure 16] It is a flowchart showing the remaining part of the process example shown in FIG. 15. [Figure 17] (A) and (B) are schematic diagrams for explaining an example of an absolute error correction process in the second embodiment. [Figure 18]It is a block diagram showing a configuration example of a distance measurement system according to a third embodiment. [Figure 19] It is a schematic diagram showing an example of output results of a dividing line detection task and an object recognition task executed by an object detection unit according to a third embodiment. [Figure 20] It is a diagram for explaining an example of the positional relationship between the camera mounting position and the road surface in the third embodiment. [Figure 21] It is a schematic diagram showing an example of a scene in which two signs with known object sizes are detected on a road. [Figure 22] It is a block diagram showing a configuration of a modified example of a scaling distance measurement unit. [Figure 23] (A) to (D) are schematic diagrams schematically showing processing examples of lane width detection and lane center detection executed by a lane analysis unit according to a third embodiment. [Figure 24] It is a schematic diagram comparing the positions indicating the lane width in a captured image with respect to the presence or absence of a roll angle. [Figure 25] It is a flowchart showing an example of a roll angle estimation process executed by a roll angle estimation unit according to a third embodiment. [Figure 26] (A) to (C) are schematic diagrams for explaining each processing example in a roll angle estimation process according to a third embodiment. [Figure 27] It is a flowchart showing an example of an estimation process of coordinates of lane width data executed by a ground contact position estimation unit according to a third embodiment. [Figure 28] It is a schematic diagram showing an example of coordinates of a ground contact position of a distance measurement target set by a ground contact position estimation unit according to a third embodiment. [[ID=3l]] [Figure 29] It is a flowchart showing an example of an estimation process of a distance to a distance measurement target executed by an object distance calculation unit according to a third embodiment. [Figure 30] It is a schematic diagram showing an example of each lane width data according to a third embodiment. [[ID=3q]]

MODE FOR CARRYING OUT THE INVENTION

[0010] <First Embodiment> A first embodiment of the present invention will be described in detail below with reference to the figures. However, the present invention is not limited to the following embodiments. In each figure, the same reference numeral is used for the same member or element, and redundant explanations are omitted or simplified.

[0011] In the following description, a route generation device (electronic device) equipped with a camera will be used as an example of the route generation device of the present invention, but the application of the present invention is not limited to this. Furthermore, in explanations that refer to diagrams, the same reference numeral will be used for parts that represent the same component, even if the diagram numbers differ, and redundant explanations will be avoided as much as possible.

[0012] Figure 1 is a schematic diagram showing an example configuration of a vehicle 100 according to the first embodiment. The vehicle 100 is an example of a mobile body comprising an imaging device 110, a radar device 120, a path generation ECU 130, a vehicle control ECU 140, and a group of measuring instruments 160. The vehicle 100 further comprises a drive unit 170 as a drive control means for driving the vehicle 100, a memory 180, and a memory 190. The drive unit 170, the memory 180, and the memory 190 will be explained using Figure 2. The vehicle 100 comprises an imaging device 110, a radar device 120, a path generation ECU 130, a vehicle control ECU 140, and a group of measuring instruments 160.

[0013] The imaging device 110 and the route generation ECU 130, etc., constitute the route generation device 150. The vehicle 100 is capable of accommodating a driver 101, and when the vehicle is in motion, the driver 101 faces forward (direction of travel) while inside the vehicle 100. The driver 101 can control the vehicle's operation by operating the steering wheel, accelerator pedal, brake pedal, and other control components of the vehicle 100. The vehicle 100 may also have an autonomous driving function, or it may be remotely controllable from the outside.

[0014] The imaging device 110 is positioned to photograph the area in front of the vehicle 100 (the normal direction of travel). As shown in Figure 1, the imaging device 110 is positioned, for example, near the upper edge of the windshield of the vehicle 100, and photographs an area within a predetermined angular range (hereinafter referred to as the shooting angle) toward the front of the vehicle 100. The imaging device 110 may also be positioned to photograph the area behind the vehicle 100 (opposite to the normal direction of travel, in the reverse direction), or to photograph the side. Multiple imaging devices 110 may be positioned on the vehicle 100.

[0015] Figure 2 is a block diagram showing an example configuration of the vehicle 100 according to the first embodiment. The imaging device 110 captures the surrounding environment of the vehicle (surrounding environment), including the road on which the vehicle 100 travels (travel road), and detects objects within the field of view of the imaging device 110. The imaging device 110 also acquires information about the detected objects (external information) and information about the distance to the detected objects (object distance information), and outputs them to the route generation ECU 130. The object distance information can be any information that can be converted to the distance from a predetermined position on the vehicle 100 to the object using a predetermined reference table or predetermined conversion coefficients and conversion formulas. For example, the distance may be assigned to a predetermined integer value and output sequentially to the route generation ECU 130.

[0016] The imaging device 110 has a sensor in which multiple pixel regions with photoelectric conversion functions are arranged in two dimensions, and is capable of acquiring the distance of an object using the image plane phase difference method. The acquisition of object distance information using the image plane phase difference method will be described later.

[0017] The radar device 120 is a detection device that detects objects by irradiating them with electromagnetic waves and receiving the reflected waves. The radar device 120 functions as a fourth distance information acquisition means that acquires distance information (fourth distance information) indicating the distance to the object in the direction of electromagnetic wave transmission, based on the time from irradiation of electromagnetic waves to reception of the reflected waves and the received intensity of the reflected waves. The radar device 120 outputs the distance information to the path generation ECU 130. The radar device 120 is assumed to be a millimeter-wave radar device that uses electromagnetic waves with wavelengths ranging from the millimeter-wave band to the submillimeter-wave band.

[0018] In this embodiment, a vehicle 100 is equipped with multiple radar devices 120, for example, radar devices 120 are mounted on the left and right sides of the front of the vehicle 100, and on the left and right sides of the rear of the vehicle 100. Each radar device 120 irradiates an area within a predetermined angular range with electromagnetic waves, and based on the time from the transmission of the electromagnetic waves to the reception of the reflected waves and the received intensity of the reflected waves, it measures the distance from each radar device 120 and generates distance information to the object. In addition to the distance from the radar device 120, the distance information may also include information on the received intensity of the reflected waves and the relative velocity of the object.

[0019] The measuring instrument group 160 includes, for example, a speed measuring instrument 161, a steering angle measuring instrument 162, and an angular velocity measuring instrument 163, and acquires vehicle information related to the vehicle's driving state, such as travel speed, steering angle, and angular velocity. The speed measuring instrument 161 is a measuring instrument that detects the travel speed of the vehicle 100. The steering angle measuring instrument 162 is a measuring instrument that detects the steering angle of the vehicle 100. The angular velocity measuring instrument 163 is a measuring instrument that detects the angular velocity of the vehicle 100 in the turning direction. Each measuring instrument outputs a measurement signal corresponding to the measured parameter as vehicle information to the route generation ECU 130.

[0020] The route generation ECU 130 is configured using logic circuits and other components to generate the vehicle 100's trajectory and related route information based on measurement signals, external information, object distance information, and distance information. The route generation ECU 130 outputs this trajectory and route information to the vehicle control ECU 140. The data processed and programs executed by the route generation ECU 130 are stored in memory 180. Here, the driving trajectory is information indicating the path (route) that vehicle 100 travels. The route information is information (including road information, etc.) necessary for vehicle 100 to travel along the route indicated by the driving trajectory.

[0021] The vehicle control ECU 140 is configured using logic circuits and the like, and controls the drive unit 170 so that the vehicle 100 travels along the path corresponding to the path information, based on the path information and vehicle information acquired from the measuring instrument group 160. The data processed and the programs executed by the vehicle control ECU 140 are stored in the memory 190.

[0022] The drive unit 170 is a drive component for driving a vehicle and includes, for example, a power unit (not shown) such as an engine or motor that generates energy to rotate the tires, and a steering unit that controls the direction of travel of the vehicle. The drive unit 170 also includes a gearbox for rotating the tires using the energy generated by the power unit, a gear control unit that controls the configuration within the gearbox, and a brake unit that performs braking.

[0023] The vehicle control ECU 140 controls the drive unit 170 so that the vehicle travels along the path corresponding to the path information, and adjusts the drive amount, braking amount, steering amount, etc. of the vehicle 100. Specifically, the vehicle control ECU 140 operates the vehicle 100 by controlling the brakes, steering, gear configuration, etc.

[0024] Furthermore, the route generation ECU 130 and the vehicle control ECU 140 may be configured with a common central processing unit (CPU) and memory for storing calculation processing programs. HMI240 is a Human Machine Interface for conveying information to the driver 101. The HMI240 includes a display that is visible when the driver 101 is in the driving position, and a display control device that generates information to be displayed on the display. The HMI240 also includes a device that outputs sound (speaker system) and a voice control device that generates voice data.

[0025] The display control unit within the HMI240 displays navigation information on the display based on the route information generated by the route generation ECU130. The voice control unit within the HMI240 also generates voice data to notify the driver 101 of the route information based on the route information, and outputs this data through the speaker system. This voice data may include, for example, data to notify the driver that an intersection to turn is approaching.

[0026] Figure 3 is a functional block diagram showing an example configuration of a route generation device 150 according to the first embodiment. The route generation device 150 includes an imaging device 110 and a route generation ECU 130. Note that some of the functional blocks shown in Figure 3 are realized by having the computer included in the route generation device 150 execute a computer program stored in memory as a storage medium. However, some or all of these may be realized in hardware. As hardware, dedicated circuits (ASICs) or processors (reconfigurable processors, DSPs) can be used.

[0027] Furthermore, the functional blocks shown in Figure 3 do not necessarily have to be housed in the same enclosure; they may be composed of separate devices connected to each other via signal paths. Furthermore, the above explanation regarding Figure 3 also applies to Figures 18 and 22.

[0028] The imaging device 110 includes an imaging optical system 301, an image sensor 302, an image processing unit 310, and an object information generation unit 320. In this embodiment, the imaging optical system 301, the image sensor 302, the image processing unit 310, and the object information generation unit 320 are arranged inside the housing (not shown) of the imaging device 110. The imaging optical system 301 is the imaging lens of the imaging device 110 and has the function of forming an image (optical image) of the subject on the image sensor 302. The imaging optical system 301 is composed of multiple lens groups. The imaging optical system 301 has an exit pupil at a predetermined distance from the image sensor 302.

[0029] The image sensor 302 is composed of CMOS (complementary metal-oxide-semiconductor) or CCD (charge-coupled device) and has a distance measurement function using an image plane phase-difference distance measurement method. The image sensor 302 has multiple pixel regions having a photoelectric conversion function arranged in two dimensions. Each pixel region has, for example, two photoelectric conversion units (first photoelectric conversion unit, second photoelectric conversion unit) that are separated in the row direction. In front of the two photoelectric conversion units of each pixel region, for example, one color filter of R, G, or B and a microlens are arranged.

[0030] The image sensor 302 converts the subject image formed on the image sensor 302 via the imaging optical system 301 into an image signal based on the subject image, and outputs it to the image processing unit 310. The image signal is a signal consisting of the output values ​​of each photoelectric conversion unit in each pixel region. The image sensor 302 outputs a first image signal based on the signal output from the first photoelectric conversion unit and a second image signal based on the signal output from the second photoelectric conversion unit separately. Alternatively, it outputs an added signal, which is the sum of the first and second image signals, and the first image signal separately to the image processing unit 310.

[0031] The image processing unit 310 generates image data containing brightness information for each of the red, green, and blue colors for each pixel, and distance image data indicating distance information for each pixel, based on the image signal supplied from the image sensor 302. The image processing unit 310 includes a development unit 311 that generates image data based on the image signal supplied from the image sensor 302, and a distance image generation unit 312 that generates distance image data based on the image signal supplied from the image sensor 302. The processing performed by these units will be described later. The image processing unit 310 outputs the image data from the development unit 311 and the distance image data from the distance image generation unit 312 to the object information generation unit 320.

[0032] The object information generation unit 320 has a recognition processing unit 321 that detects objects contained in the acquired image based on the image data from the development unit 311 and generates external information indicating information about the object. The external information indicates the position of the detected object within the image, its size such as width and height, and its area. The external information also includes information about the attributes and identification number of the detected object.

[0033] The object information generation unit 320 includes a distance information generation unit 322 that generates object distance information (first distance information) indicating the distance to an object included in the acquired image, based on external information from the recognition processing unit 321 and distance image data from the distance image generation unit 312. Here, the distance information generation unit 322 functions as a first distance information acquisition means that acquires first distance information corresponding to an object included in the image signal.

[0034] Furthermore, object distance information is linked to information about the object's identification number included in the external environment information. The object information generation unit 320 outputs external information from the recognition processing unit 321 and object distance information from the distance information generation unit 322 to the path generation ECU 130.

[0035] The image processing unit 310 and the object information generation unit 320 may be composed of one or more processors in the imaging device 110. The functions of the image processing unit 310 and the object information generation unit 320 may be realized by one or more processors executing a program read from the memory 340. The route generation ECU 130 includes a route generation unit 330. The route generation unit 330 generates route information based on external information, object distance information, and distance information acquired from the radar device 120.

[0036] Next, the structure and control of each block of the route generation device 150 will be described in detail. Figures 4(A) and 4(B) are schematic diagrams showing an example configuration of the image sensor 302 according to the first embodiment. Figure 4(A) is a top view of the image sensor 302 as seen from the direction of light incidence. The image sensor 302 is composed of multiple 2x2 pixel groups 410 arranged in a matrix. The pixel group 410 has green pixels G1 and G2 for detecting green light, red pixels R for detecting red light, and blue pixels B for detecting blue light. In the pixel group 410, green pixels G1 and G2 are arranged diagonally. Each pixel also has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412.

[0037] Figure 4(B) is a cross-sectional view of the pixel group 410 in the I-I' section of Figure 4(A). Each pixel consists of a microlens 413, a light guide layer 414, and a light receiving layer 415. The light guide layer 414 has a microlens 413 for efficiently guiding the light beam incident on the pixel to the light receiving layer 415, a color filter that allows light in a wavelength band corresponding to the color of light detected by each pixel to pass through, and wiring for image readout and pixel driving.

[0038] The light-receiving layer 415 is a photoelectric conversion unit that converts light incident through the light-guide layer 414 into electrical signals and outputs them as electrical signals. The light-receiving layer 415 has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412.

[0039] Figures 5(A) to 5(D) are schematic diagrams illustrating the relationship between subject distance and incident light in the image plane phase-difference imaging method. Figure 5(A) is a schematic diagram showing the exit pupil 501 of the imaging optical system 301, the green pixel G1 of the image sensor 302, and the light incident on each photoelectric conversion part of the green pixel G1. The image sensor 302 has multiple pixels, but for simplicity, only one green pixel G1 will be described.

[0040] The microlens 413 of the green pixel G1 is arranged such that it is optically conjugate to the exit pupil 501 and the light-receiving layer 415. As a result, the light beam that passes through the first pupil region 510, which is a partial pupil region contained within the exit pupil 501, is incident on the first photoelectric conversion unit 411. Similarly, the light beam that passes through the second pupil region 520, which is a partial pupil region, is incident on the second photoelectric conversion unit 412.

[0041] The first photoelectric conversion unit 411 of each pixel converts the received light beam into electricity and outputs a signal. A first image signal is generated from the signals output from the multiple first photoelectric conversion units 411 contained in the image sensor 302. The first image signal shows the intensity distribution of the image (referred to as image A) formed on the image sensor 302 by the light beam that mainly passed through the first pupil region 510.

[0042] The second photoelectric conversion unit 412 of each pixel converts the received light beam into electricity and outputs a signal. A second image signal is generated from the signals output from the multiple second photoelectric conversion units 412 contained in the image sensor 302. The second image signal shows the intensity distribution of the image (referred to as the B image) formed on the image sensor 302 by the light beam that mainly passed through the second pupil region 520.

[0043] The relative positional shift (hereinafter referred to as the disparity amount) between the first image signal corresponding to image A and the second image signal corresponding to image B is a quantity corresponding to the defocus amount. The relationship between the disparity amount and the defocus amount will be explained using Figures 5(B), (C), and (D).

[0044] Figures 5(B), (C), and (D) are schematic diagrams showing the image sensor 302 and the imaging optical system 301. In the figures, 511 represents the first light beam passing through the first pupil region 510, and 521 represents the light beam passing through the second pupil region 520. Figure 5(B) shows the state when the image is in focus, with the first light beam 511 and the second light beam 521 converging on the image sensor 302. At this time, the amount of disparity (positional shift) between the first image signal corresponding to image A formed by the first light beam 511 and the second image signal corresponding to image B formed by the second light beam 521 is 0.

[0045] Figure 5(C) shows a state where the image is defocused in the negative z-axis direction. At this time, the disparity between the first image signal formed by the first light beam 511 and the second image signal formed by the second light beam 521 is not zero, but has a negative value. Figure 5(D) shows the state where the image is defocused in the positive z-axis direction. At this time, the disparity between the first image signal formed by the first light beam 511 and the second image signal formed by the second light beam 521 is not zero, but has a positive value.

[0046] A comparison of Figures 5(C) and (D) shows that the direction in which parallax occurs changes depending on whether the amount of defocus is positive or negative. Furthermore, the geometric relationship shows that the amount of parallax corresponds to the amount of defocus. Therefore, as will be described later, the amount of parallax between the first image signal and the second image signal can be detected using a region-based matching method, and the amount of parallax can be converted into the amount of defocus via a predetermined conversion coefficient. In addition, by using the imaging formula of the imaging optical system 301, the amount of defocus on the image side can be converted into the distance to the object.

[0047] Furthermore, as described above, the image sensor 302 may output the summation signal (composite signal) of the first image signal and the second image signal, and the first image signal separately to the image processing unit 310. In this case, the image processing unit 310 can generate the second image signal by the difference between the summation signal (composite signal) and the first image signal, and acquire the first image signal and the second image signal, respectively.

[0048] Next, the processing performed by the image processing unit 310 will be described. Figures 6(A) and 6(B) are flowcharts illustrating examples of processing performed by the image processing unit 310 according to the first embodiment. The computer included in the path generation device 150 executes a computer program stored in memory to perform each step in the flowcharts of Figures 6(A) and 6(B). Figure 6(A) is a flowchart illustrating the operation of the development process in which the development unit 311 of the image processing unit 310 generates image data from image signals (a first image signal and a second image signal). The development process is performed in response to the reception of an image signal from the image sensor 302.

[0049] In step S601, the developing unit 311 performs a process to generate a composite image signal by combining the first image signal and the second image signal input from the image sensor 302. By combining the first image signal and the second image signal, an image signal based on the image formed by the light beam that has passed through the entire area of ​​the exit pupil 501 can be obtained. When the horizontal pixel coordinate of the image sensor 302 is x and the vertical pixel coordinate is y, the composite image signal Im(x,y) of pixel (x,y) can be expressed by the following equation (Equation 1) using the first image signal Im1(x,y) and the second image signal Im2(x,y).

[0050]

number

[0051] In step S602, the development unit 311 performs a correction process for defective pixels in the composite image signal. A defective pixel is a pixel in the image sensor 302 that cannot produce a normal signal output. The coordinates of the defective pixels are stored in memory beforehand. The development unit 311 obtains information indicating the coordinates of the defective pixels in the image sensor 302 from memory. The development unit 311 generates the composite image signal of the defective pixels using a median filter that replaces the median value of the composite image signals of the pixels surrounding the defective pixel. Alternatively, as a method of correcting the composite image signal of the defective pixels, the signal value of the defective pixel may be generated by interpolating using the signal values ​​of the pixels surrounding the defective pixel, using the coordinate information of the defective pixels that has been prepared in advance.

[0052] In step S603, the development unit 311 applies a light intensity correction process to the composite image signal to compensate for the reduction in light intensity around the field of view caused by the imaging optical system 301. The light intensity reduction characteristics (relative light intensity ratio) around the field of view caused by the imaging optical system 301 are stored in memory beforehand. As a method of light intensity correction, the relative light intensity ratio between fields of view stored in advance can be read from memory, and the composite image signal can be corrected by multiplying it by a gain such that the light intensity ratio becomes constant. For example, the development unit 311 performs light intensity correction by multiplying the composite image signal of each pixel by a gain that has the characteristic of increasing from the central pixel to the peripheral pixels of the image sensor 302.

[0053] In step S604, the developing unit 311 performs noise reduction processing on the composite image signal. As a method for reducing noise, noise reduction using a Gaussian filter can be used. In step S605, the developing unit 311 performs demosaicing on the composite image signal to acquire red (R), green (G), and blue (B) color signals for each pixel, and uses these to generate image data containing brightness information for each pixel. As a demosaicing method, a technique can be used in which the color information for each pixel is interpolated using linear interpolation for each color channel.

[0054] In step S606, the developing unit 311 performs gradation correction (gamma correction processing) using a predetermined gamma value. The image data Idc(x,y) of pixel (x,y) after gradation correction is expressed by the following equation (Equation 2) using the image data Id(x,y) of pixel (x,y) before gradation correction and the gamma value γ.

[0055]

number

[0056] In step S607, the development unit 311 performs a color space conversion process to convert the color space of the image data from the RGB color space to the YUV color space. The development unit 311 uses predetermined coefficients and a color space conversion formula (equation 3) to convert the image data corresponding to the luminance of each color (red, green, and blue) into luminance values ​​and color difference values, thereby converting the color space of the image data from the RGB color space to the YUV color space.

[0057] In Equation 3, IdcR(x,y) represents the red image data value of pixel (x,y) after tone correction. IdcG(x,y) represents the green image data value of pixel (x,y) after tone correction. IdcB(x,y) represents the blue image data value of pixel (x,y) after tone correction. Y(x,y) represents the luminance value of pixel (x,y) obtained by color space conversion. U(x,y) represents the difference (chrominance value) between the luminance value and the blue component of pixel (x,y) obtained by color space conversion.

[0058] V(x,y) represents the difference (chrominance value) between the luminance value and the red component of the pixel (x,y) obtained by color space conversion. The coefficient (ry,gy,gy) is used to find Y(x,y), and the coefficients (ru,gu,gu) and (rv,gv,bv) are used to calculate the chrominance value, respectively.

[0059]

number

[0060] In step S608, the developing unit 311 performs a correction (distortion correction) on the converted image data to suppress the effect of distortion aberration caused by the optical characteristics of the imaging optical system 301. The distortion aberration correction process is performed by geometrically transforming the image data to correct the distortion rate of the imaging optical system 301. The geometrical transformation is performed using a polynomial that generates the pixel positions before correction from the correct pixel positions without distortion aberration. If the pixel positions before correction are decimal numbers, they may be rounded to the nearest neighbor pixel, or linear interpolation may be used.

[0061] In step S609, the developing unit 311 outputs the image data to which distortion correction processing has been applied to the object information generation unit 320. This completes the developing process performed by the developing unit 311. However, the flow shown in Figure 6(A) is repeated periodically until the user issues a termination instruction (not shown).

[0062] Furthermore, if the recognition processing unit 321 can generate external information through external recognition processing based on image data obtained from the image sensor 302 before various correction processes, the development unit 311 does not need to perform all the processes shown in Figure 6(A). For example, if the recognition processing unit 321 can detect an object within the shooting angle range based on image data to which the distortion correction process in step S608 has not been applied, the process in step S608 may be omitted from the development process in Figure 6(A).

[0063] Figure 6(B) is a flowchart showing the operation of the distance image generation process performed by the distance image generation unit 312. Here, distance image data is data in which distance information corresponding to the distance from the imaging device 110 to the subject is associated with each pixel. The distance information may be a distance value D, or it may be a defocus amount ΔL or a parallax amount d used to calculate the distance value. In this embodiment, distance image data will be described as data in which a distance value D is associated with each pixel.

[0064] In step S611, the distance image generation unit 312 generates first and second luminance image signals from the input image signal. That is, the distance image generation unit 312 generates a first luminance image signal using the first image signal corresponding to image A, and generates a second luminance image signal using the second image signal corresponding to image B. The distance image generation unit 312 generates a luminance image signal by combining the image signal values ​​of the red, green, and blue pixels of each pixel group 410 using coefficients. Alternatively, the distance image generation unit 312 may generate a luminance image signal by performing demosaicing using linear interpolation and then multiplying each of the red, green, and blue channels by a predetermined coefficient and combining them.

[0065] In step S612, the distance image generation unit 312 corrects the light intensity balance between the first luminance image signal and the second luminance image signal. The light intensity balance correction is performed by multiplying at least one of the first luminance image signal and the second luminance image signal by a correction coefficient. The correction coefficient is pre-calculated after adjusting the positions of the imaging optical system 301 and the image sensor 302, providing uniform illumination, and ensuring that the luminance ratio between the obtained first luminance image signal and the second luminance image signal remains constant, and is stored in the memory 340.

[0066] The distance image generation unit 312 multiplies the correction coefficient read from memory by at least one of the first luminance image signal and the second luminance image signal to generate a first image signal and a second image signal to which light intensity balance correction has been applied.

[0067] In step S613, the distance image generation unit 312 performs noise reduction processing on the first luminance image signal and the second luminance image signal to which light intensity balance correction has been applied. The distance image generation unit 312 performs noise reduction processing by applying a low-pass filter that reduces high spatial frequency bands to each luminance image signal. Alternatively, the distance image generation unit 312 may use a band-pass filter that transmits a predetermined spatial frequency band. In this case, the effect of reducing the influence of the correction error of the light intensity balance correction performed in step S612 can be obtained.

[0068] In step S614, the distance image generation unit 312 calculates the disparity amount, which is the relative positional shift between the first luminance image signal and the second luminance image signal. The distance image generation unit 312 sets a point of interest in the first luminance image corresponding to the first luminance image signal and sets a matching region centered on the point of interest. Next, the distance image generation unit 312 sets a reference point in the second luminance image corresponding to the second luminance image signal and sets a reference region centered on the reference point.

[0069] The distance image generation unit 312 calculates the correlation between a first luminance image contained within the matching region and a second luminance image contained within the reference region while sequentially moving the reference point, and identifies the reference point with the highest correlation as the corresponding point. The distance image generation unit 312 defines the amount of relative positional shift between the point of interest and the corresponding point as the amount of disparity at the point of interest. By calculating the amount of disparity while sequentially moving the point of interest, the distance image generation unit 312 can calculate the amount of disparity at multiple pixel positions. In this manner, the distance image generation unit 312 identifies a value indicating the disparity value for each pixel and generates disparity image data, which is data indicating the disparity distribution.

[0070] Furthermore, known methods can be used as the method for calculating the correlation used by the distance image generation unit 312 to determine the amount of disparity. For example, the distance image generation unit 312 can use a method called NCC (Normalized Cross-Correlation) which evaluates the normalized cross-correlation between luminance images. Alternatively, the distance image generation unit 312 may use a method that evaluates the degree of difference as the correlation. For example, the distance image generation unit 312 can use SAD (Sum of Absolute Difference), which evaluates the sum of the absolute differences between luminance images, or SSD (Sum of Squared Difference), which evaluates the sum of the squares of the differences.

[0071] In step S615, the distance image generation unit 312 converts the disparity amount of each pixel in the disparity image data into a defocus amount to obtain the defocus amount of each pixel. Based on the disparity amount of each pixel in the disparity image data, the distance image generation unit 312 generates defocus image data that shows the defocus amount of each pixel.

[0072] The distance image generation unit 312 uses the disparity amount d(x,y) of the pixel (x,y) in the disparity image data and the conversion coefficient K to calculate the defocus amount ΔL(x,y) of the pixel (x,y) from the following equation (Equation 4). Note that the imaging optical system 301 cuts off a portion of the first light beam 511 and the second light beam 521 in the peripheral field of view due to vignetting. Therefore, the conversion coefficient K is a value that depends on the field of view (pixel position).

[0073]

number

[0074] If the imaging optical system 301 has a characteristic of field curvature in which the focal position changes between the central and peripheral fields of view, then, if the amount of field curvature is Cf, the parallax amount d(x,y) can be converted to the defocus amount ΔL(x,y) using the following equation (Equation 5). After aligning the imaging optical system 301 and the image sensor 302, the relationship between the amount of parallax and the distance to the object can be obtained by chart imaging, thereby obtaining the conversion coefficient K and the amount of field curvature Cf. In this case, the amount of field curvature Cf depends on the field of view and is given as a function of the pixel position.

[0075]

number

[0076] In step S616, the distance image generation unit 312 converts the defocus amount ΔL(x,y) of pixel (x,y) to the distance value D(x,y) to the object at pixel (x,y), and generates distance image data. The distance value D to the object can be calculated by converting the defocus amount ΔL using the imaging relationship of the imaging optical system 301. When the focal length of the imaging optical system 301 is f and the distance from the principal point on the image side to the image sensor 302 is Ipp, the defocus amount ΔL(x,y) can be converted to the distance value D(x,y) to the object using the imaging formula shown in the following equation (Equation 6).

[0077]

number

[0078] In step S617, the distance image generation unit 312 outputs the distance image data generated as described above to the object information generation unit 320. This completes the distance image data generation process performed by the distance image generation unit 312. However, the flow shown in Figure 6(B) is repeated periodically until the user issues a termination instruction (not shown). Furthermore, the parallax amount d, defocus amount ΔL, and distance value D from the principal point of the imaging optical system 301 for each pixel are values ​​that can be converted using the coefficients and conversion formulas mentioned above.

[0079] Therefore, the distance image data generated by the distance image generation unit 312 may include information representing the disparity amount d or the defocus amount ΔL for each pixel. Considering that the object information generation unit 320 calculates a representative value of the distance value D included in the object region, it is desirable to generate the distance image data based on a defocus amount that results in a symmetrical frequency distribution.

[0080] As mentioned above, in the process of calculating the disparity amount in step S614, the correlation between the first luminance image and the second luminance image is used to search for corresponding points. However, if there is a lot of noise in the first image signal (for example, noise caused by optical shot noise) or if the change in the signal value of the luminance image signal included in the matching area is small, it may not be possible to correctly evaluate the degree of correlation. In such cases, a disparity amount with a large error may be calculated compared to the correct disparity amount. If the error in the disparity amount is large, the error in the distance value D generated in step S616 will also be large.

[0081] Therefore, the distance image generation process performed by the distance image generation unit 312 may include a reliability calculation process for calculating the reliability of the disparity amount (disparity reliability). Disparity reliability is an indicator of how much error is contained in the calculated disparity amount. For example, the ratio of the standard deviation to the mean value of the signal values ​​included in the matching area can be evaluated as the disparity reliability. When the change in signal values ​​within the matching area (so-called contrast) is large, the standard deviation becomes large. When the amount of light incident on the pixel is large, the mean value becomes large. When the amount of light incident on the pixel is large, there is a lot of light shot noise. In other words, the mean value has a positive correlation with the amount of noise.

[0082] The ratio of the mean to the standard deviation (standard deviation / mean) corresponds to the ratio of the contrast magnitude to the noise level. If the contrast is sufficiently large relative to the noise level, it can be estimated that the error in calculating the parallax amount is small. In other words, the higher the parallax confidence, the smaller the error in the calculated parallax amount, and the more accurate the parallax amount can be said to be.

[0083] Therefore, in step S614, the parallax confidence level is calculated at each point of interest, and confidence data representing the certainty of the distance value for each pixel constituting the distance image data can be generated. The distance image generation unit 312 can output the confidence data to the object information generation unit 320.

[0084] Next, we will describe the process by which the object information generation unit 320 generates external information and object distance information based on image data and distance image data. Figures 7(A) and 7(B) are flowcharts showing an example of processing performed by the object information generation unit 320 according to the first embodiment, and Figures 8(A) to 8(E) are schematic diagrams showing examples of images and information in an example of processing performed by the object information generation unit 320 according to the first embodiment. The computer included in the path generation device 150 executes a computer program stored in memory, thereby performing the operation of each step in the flowcharts of Figures 7(A) and 7(B).

[0085] The recognition processing unit 321 generates external information based on the image data, including the position of an object in the image, its size (such as width and height), area information indicating its region, the type (attributes) of the object, and an identification number (ID number). The identification number is identification information for identifying the detected object and is not limited to a number. The recognition processing unit 321 detects the type of object present within the imaging angle of the imaging device 110, as well as the position and size of the object in the image, and determines whether or not the object has already been registered, and assigns an identification number.

[0086] Figure 8(A) shows an image 810 based on image data acquired by the imaging device 110 and input to the object information generation unit 320. Image 810 includes a person 801, a vehicle 802, a sign 803, a road 804, and a lane 805. The recognition processing unit 321 detects objects from image 810 and generates external information indicating the type, identification number, and region information of each object. Figure 8(B) is a schematic diagram showing the external information of each object detected from image 810 at each position in image 810 on the xy coordinate plane.

[0087] For example, external information is generated as a table, such as the one shown in Table 1. In the external information, the area of ​​an object is defined as a rectangular frame (object frame) surrounding the object. In the external information, the area information of an object is indicated by the shape of the rectangular object frame as the coordinates of the top left (x0, y0) and the bottom right (x1, y1).

[0088] [Table 1]

[0089] Figure 7(A) is a flowchart showing an example of the process by which the recognition processing unit 321 generates external information. The recognition processing unit 321 starts the process of generating external information in response to, for example, the acquisition of image data. In step S701, the recognition processing unit 321 generates image data to be used for object detection processing from the image data. That is, the recognition processing unit 321 processes the image data input from the image processing unit 310 to a size determined by the detection performance and processing time in the object detection process.

[0090] In step S702, the recognition processing unit 321 performs a process to detect objects contained in the image based on the image data, and detects the region corresponding to the object in the image and the type of object. The recognition processing unit 321 may detect multiple objects from a single image. In this case, the recognition processing unit 321 identifies the type and region for the multiple detected objects.

[0091] The recognition processing unit 321 generates external information including the position and size (horizontal width, vertical height) of the area in the image where the object is detected, and the type of object. The types of objects that the recognition processing unit 321 can detect include, for example, vehicles (passenger cars, buses, trucks), people, animals, motorcycles, signs, etc. The recognition processing unit 321 detects objects and identifies the type of detected object by comparing the outline of the object in the image with a predetermined outline pattern associated with the type of object. The types of objects that the recognition processing unit 321 can detect are not limited to those listed above, but from the viewpoint of processing speed, it is desirable to narrow down the number of object types to be detected according to the driving environment of the vehicle 100.

[0092] In step S703, the recognition processing unit 321 tracks objects for which an identification number has already been registered. The recognition processing unit 321 identifies objects for which an identification number has already been registered from among the objects detected in step S702. An object for which an identification number has been registered is, for example, an object that was detected in a previous object detection process and assigned an identification number. When an object with a registered identification number is detected, the recognition processing unit 321 associates the information about the type of object and the region acquired in step S702 with the external information corresponding to that identification number (updates the external information).

[0093] If it is determined that an object with a registered identification number does not exist in the image information, the tracking is interrupted, as it is determined that the object associated with that identification number has moved outside the field of view of the imaging device 110 (is lost). In step S704, the recognition processing unit 321 determines whether each of the objects detected in step S702 is a new object for which an identification number has not been registered. Then, it assigns a new identification number to the external information indicating the type and region of the object determined to be a new object, and registers it in the external information.

[0094] In step S705, the recognition processing unit 321 outputs the generated external information along with time information to the route generation device 150. This completes the external information generation process performed by the recognition processing unit 321. However, the flow shown in Figure 7(A) is repeated periodically until the user issues a termination instruction (not shown).

[0095] Figure 7(B) is a flowchart showing an example of the distance information generation process for each object performed by the distance information generation unit 322. Based on the external information and distance image data, the distance information generation unit 322 generates object distance information representing the distance value for each detected object. Figure 8(C) is a diagram showing an example of a distance image 820, which is generated based on the image data in Figure 8(A) and associates the distance image data with the image 810. In the distance image 820 of Figure 8(C), the distance information is shown by the intensity of the color, with darker colors indicating closer objects and lighter colors indicating farther objects.

[0096] In step S711, the distance information generation unit 322 counts the number of objects N detected by the recognition processing unit 321 and calculates the total number of detected objects Nmax. In step S712, the distance information generation unit 322 sets N to 1 (initialization process). The processes from step S713 onward are executed sequentially for each object shown in the external information. Assume that the processes from step S713 to step S716 are executed in order of increasing identification number in the external information.

[0097] In step S713, the distance information generation unit 322 identifies a rectangular region on the distance image 820 that corresponds to the region (object frame) on the image 810 of the Nth object included in the external information. The distance information generation unit 322 sets a frame (object frame) on the distance image 820 that shows the outline of the corresponding region. Figure 8(D) is a schematic diagram in which frames showing the outline of the region set on the distance image 820 are superimposed for each object detected from the image 810. As shown in Figure 8(D), the distance information generation unit 322 sets an object frame 821 corresponding to a person 801, an object frame 822 corresponding to a vehicle 802, and an object frame 823 corresponding to a sign 803 on the distance image 820.

[0098] In step S714, the distance information generation unit 322 generates a frequency distribution of distance information for pixels contained within the rectangular region of the distance image 820 corresponding to the Nth object. If the information associated with each pixel of the distance image data is the distance value D, the intervals of the frequency distribution are set so that the reciprocals of the distances are equally spaced.

[0099] Furthermore, if each pixel of the distance image data is associated with a defocus amount or disparity amount, it is desirable to divide the frequency distribution intervals into equal intervals. In step S715, the distance information generation unit 322 uses the most frequently occurring distance information from the frequency distribution as object distance information indicating the distance of the Nth object.

[0100] Alternatively, the average of the distance values ​​contained within the region may be calculated and used as object distance information. When calculating the average, a weighted average can be used using confidence data. By setting a larger weight for each pixel as the confidence level of the distance value increases, the object distance can be calculated with greater accuracy.

[0101] Furthermore, in order to facilitate path generation in the path generation process described later, it is desirable that the object distance information be information indicating the distance from a predetermined position on the vehicle 100 to the object. When using a distance value D as the distance information, since the distance value D indicates the distance from the image sensor 302 to the object, the most frequent value can be offset by a predetermined amount to obtain information indicating the distance from a predetermined position on the vehicle 100 (for example, the front end) to the object. When using a defocus amount ΔL as the distance information, it is converted to the distance from the image sensor 302 using equation 6, and then offset by a predetermined amount to obtain information indicating the distance from a predetermined position on the vehicle 100 to the object.

[0102] In step S716, the distance information generation unit 322 determines whether N+1 is greater than the number of detected objects Nmax. If N+1 is less than the number of detected objects Nmax (the answer is No in step S716), then in step S717, the distance information generation unit 322 sets N to N+1, and the process returns to step S713. That is, object distance information is extracted for the next object (the (N+1)th object). If N+1 is greater than the number of detected objects Nmax in step S716 (the answer is Yes in step S716), the process proceeds to step S718.

[0103] In step S718, the distance information generation unit 322 outputs object distance information for each of the Nmax objects, along with time information, to the path generation unit 330, and then terminates processing. This information is stored in memory 180. Distance information from the radar device 120 is also stored in memory 180 along with time information. The flow shown in Figure 7(B) is repeated periodically until the user issues a termination instruction (not shown).

[0104] The object distance information generation process described above generates object distance information for each object included in the external information. In particular, by statistically determining the object distance information from the distance information contained in the region of the distance image 820 corresponding to the object detected in image 810, it is possible to suppress variations in pixel-by-pixel distance information caused by noise and calculation accuracy. Therefore, it is possible to obtain information indicating the distance of an object with higher accuracy. As mentioned above, the method for statistically determining the distance information is to take the most frequent distance information, mean, median, etc., from the distribution of distance information, and various methods can be employed.

[0105] Next, we will explain the process (route generation process) that generates route information, which is performed by the route generation unit 330 of the route generation ECU 130. Route information includes information such as the direction of travel and speed of the vehicle. Route information can also be said to be operation plan information. The route generation unit 330 outputs the route information to the vehicle control ECU 140. Based on the route information, the vehicle control ECU 140 controls the direction of travel and speed of the vehicle by controlling the drive unit 170.

[0106] In this embodiment, the route generation unit 330 generates route information so that the vehicle 100 follows another vehicle (a vehicle ahead) if there is another vehicle (a vehicle ahead) in the direction of travel of the vehicle 100. Furthermore, the route generation unit 330 generates route information so that the vehicle 100 takes evasive action to avoid colliding with an object.

[0107] Figure 9 is a flowchart showing an example of route generation processing performed by the route generation unit 330 according to the first embodiment. Note that the computer included in the route generation device 150 executes a computer program stored in memory, thereby performing each step of the flowchart in Figure 9.

[0108] The route generation unit 330, for example, when instructed by the user to start the route generation process, starts the flow shown in Figure 9 and generates route information for the vehicle 100 based on external information, object distance information, and distance information generated by the radar device 120. The route generation ECU 130 reads the external information, object distance information, and distance information generated by the radar device 120 for each time point from the memory 180 of the route generation device 150, for example, and processes it.

[0109] In step S901, the path generation unit 330 detects objects on the planned driving path of the vehicle 100 from external information and object distance information. The path generation unit 330 determines objects on the driving path by comparing the direction of travel planned by the vehicle 100 with the position and type of objects included in the external information. The direction of travel planned by the vehicle 100 is determined based on information related to the control of the vehicle 100 (steering angle, speed, etc.) obtained from the vehicle control ECU 140. If no objects are detected on the driving path, the path generation unit 330 determines "no objects".

[0110] For example, suppose the imaging device 110 acquires the image 810 shown in Figure 8(A). If the path generation unit 330 determines from the control information of the vehicle 100 acquired from the vehicle control ECU 140 that the vehicle 100 is moving in the direction along lane 805, the path generation unit 330 detects the vehicle 802 as an object on the travel path.

[0111] In steps S902 and S903, the path generation unit 330 determines whether to generate path information for follow-the-object driving or path information for avoidance driving, based on the distance between the vehicle 100 and the object on the driving path and the speed Vc of the vehicle 100.

[0112] In step S902, the path generation unit 330 determines whether the distance between the vehicle 100 and the object on the travel path is shorter than the threshold Dth. The threshold Dth is expressed as a function of the vehicle's travel speed Vc. The higher the travel speed, the larger the threshold Dth. If the path generation unit 330 determines that the distance between the vehicle 100 and the object on the travel path is shorter than the threshold Dth (step S902 Yes), the process proceeds to step S903. If the path generation unit 330 determines that the distance between the vehicle 100 and the object on the travel path is greater than or equal to the threshold Dth (step S902 No), the process proceeds to step S908.

[0113] In step S903, the route generation unit 330 determines whether the relative speed between the vehicle 100 and the object on the travel path is a positive value. The route generation unit 330 obtains the identification number of the object on the travel path from external information and obtains object distance information for the object on the travel path at each time from external information obtained from the present time up to a predetermined time ago. The route generation unit 330 calculates the relative speed between the vehicle 100 and the object on the travel path from the object distance information for the period up to the predetermined time ago that has been obtained. If the relative speed (vehicle speed 100 - speed of the object on the travel path) is a positive value, it indicates that the vehicle 100 and the object on the travel path are approaching each other.

[0114] If the path generation unit 330 determines that the relative speed between the vehicle 100 and the object on the travel path is a positive value (step S903 Yes), the process proceeds to step S904. If the path generation unit 330 determines that the relative speed between the vehicle 100 and the object on the travel path is not a positive value (step S903 No), the process proceeds to step S908.

[0115] If the process proceeds to step S904, route information for executing evasive action is generated. If the process proceeds to step S908, route information for performing follow-up driving that maintains a distance from the vehicle in front is generated.

[0116] In other words, the path generation unit 330 performs an evasive action when the distance between the vehicle 100 and the object on the travel path is shorter than the threshold Dth, and the relative speed between the vehicle 100 and the object on the travel path is a positive value. On the other hand, the path generation unit 330 determines to follow the object if the distance between the vehicle 100 and the object on the travel path is greater than or equal to the threshold Dth. Alternatively, the path generation unit 330 determines to follow the object if the distance between the vehicle 100 and the object on the travel path is shorter than the threshold Dth, and the relative speed between the vehicle 100 and the object on the travel path is zero or a negative value (moving away).

[0117] If the distance between vehicle 100 and an object on the travel path is shorter than a threshold Dth calculated from the speed of vehicle 100, and the relative speed between vehicle 100 and the object on the travel path is a positive value, then there is a high probability that vehicle 100 will collide with the object on the travel path. Therefore, the path generation unit 330 generates path information to take evasive action. Otherwise, the path generation unit 330 takes a follow-the-object driving stance. Furthermore, the above determination may also include a determination of whether the detected object on the travel path is a moving object (such as a car or motorcycle).

[0118] In step S904, the path generation unit 330 starts the process of generating path information for executing an avoidance action. In step S905, the path generation unit 330 acquires information regarding avoidance spaces. The path generation unit 330 acquires distance information from the radar device 120, including the distance to objects to the sides and rear of the vehicle 100, and predicted distance information to those objects.

[0119] The path generation unit 330 acquires information indicating the direction and size of the space around the vehicle 100 in which the vehicle 100 can move, based on distance information (fourth distance information) acquired from the radar device 120, the speed of the vehicle 100, and information indicating the size of the vehicle 100. In this embodiment, distance information (fourth distance information) acquired from the radar device 120 is used for avoidance space, but it may also be used to generate integrated distance information to an object.

[0120] In step S906, the path generation unit 330 sets path information for avoidance action based on information indicating the direction and size of the space in which the vehicle 100 can move, external information, and object distance information. The path information for avoidance action is, for example, information to change the vehicle 100's course to the right, for example, while decelerating, if there is a space to avoid to the right of the vehicle 100.

[0121] In step S907, the route generation unit 330 outputs route information to the vehicle control ECU 140. The vehicle control ECU 140 determines parameters to control the drive unit 170 based on the route information so that the vehicle 100 travels along the route indicated in the acquired route information, and controls the drive unit 170. Specifically, the vehicle control ECU 140 determines the steering angle, accelerator control value, brake control value, control signal for gear engagement, and lamp illumination control signal based on the route information.

[0122] Furthermore, step S907 functions as a route generation step (route generation means) that generates route information based on distance information. Meanwhile, in step S908, the route generation unit 330 starts the process of generating route information for performing follow-up operation.

[0123] In step S909, the route generation unit 330 generates route information so that the vehicle 100 follows an object (a preceding vehicle) on its travel path. Specifically, the route generation unit 330 generates route information so that the distance between the vehicle 100 and the preceding vehicle (inter-vehicle distance) remains within a predetermined range. For example, if the relative speed between the vehicle 100 and the preceding vehicle is zero or a negative value, or if the inter-vehicle distance is greater than or equal to a predetermined distance, the route generation unit 330 generates route information so that the vehicle 100 maintains a straight direction of travel while accelerating and decelerating to maintain the predetermined inter-vehicle distance.

[0124] The route generation unit 330 generates route information so that the vehicle 100's travel speed does not exceed a predetermined value (for example, the legal speed limit on the road the vehicle 100 is traveling on, or a set travel speed based on instructions from the driver 101). After step S909, the process proceeds to step S907, in which the vehicle control ECU 140 controls the drive unit 170 based on the generated route information.

[0125] Next, in step S910, it is determined whether the user has issued an instruction to terminate the route generation process. If yes, the route generation unit 330 terminates the route information generation process. If no, the process returns to step S901 and the route information generation process is repeated.

[0126] According to the control described above, by statistically processing distance information within a frame based on the position and size of objects in the image, the effects of sensor noise and local distance errors caused by high-luminance reflections from the subject can be reduced, and the distance value to each object can be calculated with high accuracy. Furthermore, since the distance value to each object can be calculated with high accuracy, the route generation ECU 130 can calculate the route of vehicle 100 with high accuracy, enabling vehicle 100 to drive more stably.

[0127] <Example 1> In the process described above, the object's region was shown as a rectangular area (object frame) encompassing the object. However, the object's region may also be defined as a region with the shape of the object, bounded by the outer perimeter of the object within the image. In that case, in step S702, the recognition processing unit 321 stores the region in the image 810 where the object exists as the object's region in the external information. For example, the recognition processing unit 321 can divide the image into regions for each object by identifying attributes for each pixel of the image data.

[0128] Figure 8(E) is a schematic diagram showing an example in which the recognition processing unit 321 performs region division for each object and superimposes the results onto the image 810. Region 831 represents the region of person 801, region 832 represents the region of vehicle 802, and region 833 represents the region of sign 803. Furthermore, region 834 represents the region of road 804, and region 835 represents the region of lane 805.

[0129] In this case, in steps S713 and S714, the distance information generation unit 322 can calculate, for example, the frequency distribution of distance values ​​contained within each region shown in Figure 8(E) for each object.

[0130] By defining the object's region in this way, distance information from non-object elements such as the background becomes less likely to be included within the region. In other words, the distribution of distance information within the region becomes more accurately reflected in the object's distance information. Therefore, the influence of non-object elements such as the background and foreground can be reduced, allowing for more accurate calculation of the object's distance value.

[0131] <Modification 2> In this embodiment, the imaging device 110 sequentially outputs image information and distance image information. Furthermore, the recognition processing unit 321 sequentially generates external information using the sequentially received image information. The external information includes the identification number of an object, and if an object with the same identification number is detected at a certain time T0 and time T1, it is possible to determine the time changes of the distance information and detected size of that object. Accordingly, in the modified example 2, the distance information generation unit 322 calculates the average of the distance values ​​D of objects with the same identification number within a predetermined time range. This reduces distance variation in the time direction.

[0132] Figure 10 is a flowchart showing an example of object distance information generation processing performed by the distance information generation unit 322 in modified example 2. The computer included in the path generation device 150 executes a computer program stored in memory to perform each step in the flowchart of Figure 10. For each process in this flowchart, the processes indicated with the same numbers as those shown in Figure 7(B) are the same as those described in Figure 7(B), and therefore their explanation is omitted.

[0133] In step S1000, the distance information generation unit 322 selects the most frequently occurring distance information from the frequency distribution generated in step S714 and uses it as object distance information indicating the distance to the Nth object. The distance information generation unit 322 then stores (remembers) the object distance information in the memory 340 along with the identification number and time.

[0134] In step S1001, the distance information generation unit 322 obtains a history of object distance information with the same identification number as the Nth object from the object distance information stored in the memory 340. The distance information generation unit 322 obtains object distance information with the same identification number corresponding to a predetermined time before the time corresponding to the latest object distance information. Figure 11 is a schematic diagram illustrating an example of the time change of object distance information for an object with the same identification number as the Nth object. The horizontal axis shows time, and the vertical axis shows object distance information (distance value D). Time t0 indicates the time when the latest distance value D was obtained.

[0135] In step S1002, the distance information generation unit 322 calculates the average value of object distance information included in a time range from the time the latest object distance information was acquired to a predetermined time before, based on the history of object distance information of objects with the same identification number as the acquired Nth object. For example, in Figure 11, the distance information generation unit 322 calculates the average value of the distance values ​​of four points included in the predetermined time range ΔT.

[0136] As described above, by using the identification number included in the external information to obtain a history of object distance information (distance value) for the same object and taking a time average, it is possible to suppress variability. Even if the road on which vehicle 100 travels changes (for example, curves, slopes, rough roads with many bumps, etc.), it is possible to track the same object and calculate the average value in the time direction.

[0137] Therefore, while reducing the effects of changes in the driving environment, it is possible to reduce the variation in distance values ​​due to noise such as optical shot noise contained in the image signal and calculate the distance value of an object with greater accuracy. In the modified example 2, in order to obtain a similar effect, when acquiring object distance information, it is also possible to acquire object distance information that has been time-averaged to some extent through a low-pass filter.

[0138] <Variation 3> In the modified example 2 described above, variability was suppressed by averaging the object distance information for objects with the same identification number over time. When averaging the history of object distance information within a predetermined time range, the number of samples used for averaging can be increased by extending the time range. Therefore, the variability in distance values ​​from vehicle 100 to objects can be further reduced.

[0139] However, if the distance from vehicle 100 to the object changes within a predetermined time range, the distance change will be included in the average, which may prevent the accurate estimation of the distance value from vehicle 100 to the object. In Modification 3, by performing a weighted average of object distance information using the size of objects with the same identification number, it becomes possible to obtain the distance between vehicle 100 and the object with higher accuracy.

[0140] Figure 12 is a flowchart showing an example of object distance information generation processing performed by the distance information generation unit 322 in modified example 3. The computer included in the path generation device 150 executes a computer program stored in memory to perform each step of the flowchart in Figure 12. For each process in this flowchart, the processes with the same numbers as those shown in Figures 7(B) and 10 are the same as the processes described above, and therefore their explanation is omitted.

[0141] In step S1000, the distance information generation unit 322 selects the most frequently occurring distance information from the frequency distribution and uses it as object distance information indicating the distance to the Nth object. The distance information generation unit 322 then stores the object distance information, along with the identification number and time, in the memory 340.

[0142] In step S1201, the distance information generation unit 322 retrieves from memory 340 the history of object distance information for objects with the same identification number as the Nth object, and the history of information indicating the size of the object. The information indicating the size of the object is obtained from the information indicating the object frame stored in the external information. For example, the width (x1-x0) from the top-left coordinate (x0, y0) and bottom-right coordinate (x1, y1) is used as information indicating the size of the object.

[0143] In step S1202, the distance information generation unit 322 uses information indicating the size of objects with the same identification number as the Nth object to perform a weighted averaging process on object distance information for the same identification number corresponding to a predetermined time period prior to the time corresponding to the latest object distance information. The weight coefficient for each time period is determined using the object size at the corresponding time period.

[0144] Figures 13(A) to 13(D) are schematic diagrams illustrating weighted averaging. Figure 13(A) is a schematic diagram showing image 1300 based on image data acquired by the imaging device 110 at time t1, prior to time t0, which corresponds to the latest object distance information. Image 1300 includes vehicle 1301. Frame 1311 shows the object frame of vehicle 1301 determined from image 1300.

[0145] Figure 13(B) is a schematic diagram showing image 1310 based on image data acquired by the imaging device 110 at time t0. Image 1310 includes the vehicle 1301, similar to image 1300. Frame 1312 indicates the object frame corresponding to the vehicle 1301 in image 1310. At time t0, the size of the vehicle 1301 in image 1300 is larger than the size in image 1310 acquired at time t1. Object frame 1312 is larger than object frame 1311.

[0146] Since the size of an object in an image is proportional to the horizontal magnification of the imaging optical system 301, the distance between the object and the vehicle 100 is directly proportional to the reciprocal of the object's size in the image information. By comparing the object's size in the image information at different time points, for example, an increase in size indicates that the distance between the object and the vehicle 100 has decreased, while a decrease in size indicates that the distance between the object and the vehicle 100 has increased. Furthermore, a small change in size indicates a small change in the distance between the object and the vehicle 100.

[0147] As an example, let's assume that we obtain distance information for vehicle 1301 as the Nth object. Figure 13(C) is a schematic diagram showing the time change of object distance information for an object with the same identification number as vehicle 1301 (the Nth object). Figure 13(D) is a schematic diagram showing the time change of the reciprocal of the size information (width) for an object with the same identification number as vehicle 1301 (the Nth object).

[0148] In step S1202, the distance information generation unit 322 compares the reciprocal of the object's size (width) at time t0 with the reciprocal of the object's size (width) at each time from time t0 to a predetermined time prior. The distance information generation unit 322 determines the weighting coefficient such that the larger the absolute value of the difference in the reciprocals of the object's size (width), the smaller the weighting coefficient becomes. Note that the relationship between the reciprocal of the object's size (width) and the weighting coefficient is not limited to the example above; for example, the weighting coefficient may be determined according to the ratio of the reciprocal of the object's size (width) at each time to the reciprocal of the object's size (width) at time t0.

[0149] In step S1202, the distance information generation unit 322 weights and averages the object distance information using weighting coefficients to obtain the object distance information of the vehicle 1301 at time t0. According to the processing in Modification 3, by weighting and averaging object distance information using weight coefficients determined using the size of the object in the image, the estimation error of distance values ​​caused by relative distance changes from the vehicle 100 to the object can be reduced.

[0150] <Second Embodiment> In the first embodiment, the distance to an object in an image (object distance information) was obtained with high accuracy by statistically processing distance data acquired for the object using the image plane phase-difference method. In the second embodiment, the distance to an object can be obtained with even higher accuracy by combining the image plane phase-difference method for calculating distance and the image recognition method for calculating distance. Hereafter, the distance calculation method and the distance calculation process based on that method will be referred to as "distance measurement".

[0151] A second embodiment of the present invention will be described in detail below with reference to the figures. The following processes can be performed by any one or a combination of the processors constituting the image processing unit 310 and object information generation unit 320 of the imaging device 110, and the path generation ECU 130.

[0152] In the second embodiment, the path generation device 150 corrects the measured distance value by combining imaging plane phase difference distance measurement and distance measurement using image recognition.

[0153] Image plane phase difference distance measurement is a distance measurement method using the image plane phase difference method described in the first embodiment. Distance measurement using image recognition includes distance measurement that calculates the distance from the width of an object detected based on object recognition (object width distance measurement) and distance measurement that calculates the distance based on information about the object's ground contact position (ground contact position distance measurement).

[0154] Object width distance measurement calculates the distance to an object by utilizing the fact that the fewer pixels an object has in its width on an image, the farther the object is, and the more pixels an object has in its width, the closer the object is. It is also possible to use other parameters indicating the size of the object on the image (e.g., the size of the object's frame), such as height and diagonal direction, in the distance calculation.

[0155] In ground contact distance measurement, it is assumed that the object is in contact with, for example, the road surface, and the distance from vehicle 100 to the object is calculated based on the distance between the ground contact line on the object's image (which would be the bottom edge if the road surface is below in the image) and the vanishing point on the image. The closer the ground contact line is to the vanishing point, the greater the distance the object is from vehicle 100, and the farther the ground contact line is from the vanishing point, the closer the object is from vehicle 100.

[0156] This section explains the characteristics of errors in image plane phase difference distance measurement, object width distance measurement, and ground contact position distance measurement. Common to all of these methods, relative error and absolute error are defined as follows: Relative error is defined as the quantity equivalent to the standard deviation for a sufficient number of samples when there is no change in relative distance. Absolute error is defined as the quantity equivalent to the difference between the mean value and the true value for a sufficient number of samples when there is no change in relative distance.

[0157] The relative error in image-plane phase-difference ranging is primarily caused by parallax errors resulting from block matching due to pixel value variations caused by sensor noise. Since this does not change depending on the parallax value, when converted to distance, the relative error basically worsens in proportion to the square of the distance. Absolute errors arise from optical system aberrations, assembly errors, and fluctuations due to heat and vibration. While it is possible to correct for each of these factors, if correction is not performed considering the computational load, a significant amount of error may remain.

[0158] The relative error in object width measurement depends on the resolution and recognition accuracy of the object in the image. In object width measurement, the width of the detected object in the image cannot be converted to distance unless the actual object width (the actual width of the object, expressed in units such as meters) is known. Therefore, the actual object width must be determined in some way, and both the absolute and relative errors depend on that actual object width. Since the relative error is proportional to the distance, the relative error may be smaller than that of image-plane phase-difference distance measurement at long distances.

[0159] The relative error in ground contact distance measurement depends on the accuracy of recognizing the ground contact line on the object's image and the image resolution. Since image resolution is the resolution used to measure the distance between the vanishing point and the ground contact line, higher resolution images allow for more accurate measurements even at long distances. Furthermore, when the road surface extends downwards, the estimation error of the pitch angle in the optical axis direction becomes the distance measurement error.

[0160] When an image acquisition device is mounted on a moving object, the pitch angle fluctuates with each image frame due to the acceleration of the movement and the road surface conditions. In this case, the pitch angle error is a relative error. A pitch angle error that always occurs at a constant value due to the installation conditions and the tilt of the moving object itself is an absolute error. As will be discussed later, the pitch angle can be estimated using vanishing point information and movement information to reduce the error. The relative error is proportional to the distance and is equivalent to the width of the object, but the amount of error is larger than that of the width of the object because it is affected by the pitch angle estimation. Distance measurement based on the object's ground contact position requires that the object is in contact with the ground, so there is a problem that distance cannot be measured if the lower end is not in contact with the ground, such as with traffic signals or signs.

[0161] Figure 14 is a flowchart showing an example of the process for acquiring distance measurement values ​​(object distance information) in the second embodiment. Note that the computer included in the path generation device 150 executes a computer program stored in memory, thereby performing the operation of each step in the flowchart of Figure 14.

[0162] Data D1401 is data input to the distance information generation unit 322. As described in the first embodiment, the distance information generation unit 322 receives external information about an object obtained by image recognition from the image captured by the image sensor 302 (object identification number, object type (attributes), and size of the area corresponding to the object). Data D1401 may also include results from other image recognition processes, and may include information that indicates the pixel position on the image, such as image coordinates representing the image range of the recognized image or information about the object's region obtained by semantic region segmentation technology. It may also include other image recognition result information.

[0163] Furthermore, the distance information generation unit 322 receives distance image data as data D1401, which represents the results (distance information for each pixel) obtained by calculation using the image plane phase difference method. In this embodiment, in addition to this data, information regarding the focal length f of the imaging optical system 301, the moving speed of the vehicle 100, and the installation position of the imaging device 110 are also input to the distance information generation unit 322 as data D1401. Data D1401 is a data set that combines all of this information.

[0164] In the following steps, recognition processing may be performed on multiple objects simultaneously. However, if the process involves saving and referencing time-series data information, it will only be performed on objects that are recognized as the same object. In other words, they have the same identification number as input data.

[0165] In step S1401, the distance information generation unit 322 acquires a measured distance D1 by ground position distance measurement and outputs it as data D1402. The measured distance D1 indicates the distance between the vehicle 100 (imaging device 110) and the target object calculated by ground position distance measurement. Data D1402 is information indicating the measured distance D1 (second distance information) calculated by ground position distance measurement in step S1401. Here, step S1401 functions as a second distance information acquisition step (second distance information acquisition means) that acquires second distance information based on information of the edge position of the object included in the image signal.

[0166] The distance information generation unit 322 obtains the pixel position of the ground contact point on the image using image coordinates that represent the image range of the recognized object contained in the data D1401. The outline of the distance measurement process when the optical axis is set to be parallel to the road surface and at height H, and the distance between the vanishing point on the image and the ground contact line is Hs pixels (or subpixel units) is described below. If the image is in the central projection method (or an image corrected to the central projection method) with a focal length f and pixel size Ps, the distance measurement value (distance) D1 can be expressed as follows using the following equation (Equation 7).

[0167]

number

[0168] Equation 7 assumes that the line is parallel to the road surface. However, as mentioned earlier, if there is an error in the pitch angle of the moving object, the vanishing point will be in a different position than assumed, resulting in an error in the value of Hs and consequently a distance error. Furthermore, if the recognition accuracy is poor, Hs will be recognized as being in a different position from the actual grounding line, resulting in a similar distance error.

[0169] In step S1402, the distance information generation unit 322 acquires a distance value D2 by imaging plane phase difference distance measurement and outputs it as data D1404. Data D1404 is information indicating the distance value D2 (first distance information) calculated in step S1402 by imaging plane phase difference distance measurement. That is, the first distance information is distance information acquired by the phase difference distance measurement method based on the signals from the first and second photoelectric conversion units described above. Alternatively, it is distance information acquired by the phase difference distance measurement method based on two image signals from a stereo camera. Here, step S1402 functions as a first distance information acquisition step (first distance information acquisition means) that acquires first distance information corresponding to an object included in the image signal.

[0170] As shown in the first embodiment, the distance information generation unit 322 can acquire distance information (distance value) of a target object based on distance image data and external information. For example, suppose that the input data D1401 is distance image data in which the distance information of each pixel is indicated by the amount of defocus, and external information in which the region of the object is indicated by a frame.

[0171] At this time, the distance information generation unit 322 can obtain the distance between the vehicle 100 and the target object (object distance information) from the imaging formula using the most frequent value of the amount of defocus included in the object frame of the target object and the focal length f. The obtained distance is acquired as the measured distance value D2. Note that the input data D1401 may be the distance value itself, or other data from the calculation process may be entered.

[0172] In step S1403, the distance information generation unit 322 obtains the width (object width) Ws of the target object in the image. The object width Ws is expressed in terms of the number of pixels. The number of objects may also be expressed in subpixel units. The distance information generation unit 322 measures the object width Ws from the image recognition result. For example, if the external information included in data D1401 includes information indicating the object frame of each object in the image, the width of the object frame corresponding to the target object may be used as the object width Ws. Furthermore, it does not have to be the width of the object; it may also be the height, and either with a larger number of pixels may be chosen, or both may be used to improve robustness.

[0173] If the object's attribute information is known, it can be taken into consideration when making a decision. Also, for example, if the object is a vehicle and is outside the center of the field of view, it is highly likely that the width of the object includes the side of the vehicle. In such cases, it is better to select the height. When looking at changes over time, it is also possible to consider fluctuations in the ratio of height to width and select information that is stable when compared with distance changes. The distance information generation unit 322 uses data D1403 to represent the information indicating the object width Ws.

[0174] In step S1404, the distance information generation unit 322 uses the object width Ws from data D1403 and either the distance value D1 obtained by ground contact position distance measurement from data D1402 or the distance value D2 obtained by imaging plane phase difference distance measurement from data D1404, or both. Then it calculates the actual object width W. The actual object width W is information that indicates the width of the target object in a unit system that indicates length (such as meters).

[0175] As described above, the actual object width W can be determined using either or both of the distance values ​​D1 and D2 for the object width Ws, but it is desirable to choose the distance value with the smallest absolute error between D1 and D2. When using the distance value D1, the actual object width W can be expressed as follows using the following equation (Equation 8).

[0176]

number

[0177] Furthermore, the actual object width W may be determined based on information indicating the type of object included in the external information of the input data D1401. That is, integrated distance information may be generated based on the type of object. For example, if the type of object is a passenger car, the actual object width W can be set to a predetermined value, for example, 1.7m. However, if the actual object width W is determined according to the type of object, it will strictly differ for each object, and the difference will be an absolute error. The distance information generation unit 322 outputs information indicating the actual object width W as data D1405.

[0178] In step S1405, the distance information generation unit 322 obtains a distance value D3 using the actual object width W and the object width Ws. Data D1406 (third distance information) is information indicating the distance value D3 based on the object width calculated in step S1405. Here, step S1405 functions as a third distance information acquisition step (third distance information acquisition means) that acquires third distance information based on information about the size of the object (width or height of the object) included in the image signal. The process performed in step S1405 is the reverse process of step S1404. The distance value D3 can be expressed as follows using the following equation (equation 9).

[0179]

number

[0180] Furthermore, the step group C1101, consisting of steps S1403, S1404, and S1405, is for measuring the width of the object. In step S1406, the distance information generation unit 322 integrates the measured distance D1 from data D1402, the measured distance D3 from data D1406, and the measured distance D2 from data D1404 to obtain the distance value D to the recognized object.

[0181] Furthermore, it is not necessary to combine all of the first to third distance information. It is sufficient to generate integrated distance information by combining at least two of the first to third distance information. In this embodiment, both the second distance information acquisition means and the third distance information acquisition means are provided, but it is sufficient to provide at least one of them. Here, step S1406 functions as a distance information integration step (distance information integration means) that combines at least two of the first to third distance information to generate integrated distance information.

[0182] The integration process can be, for example, a process of selecting one of the distance measurement values ​​D1 (first distance information), D2 (second distance information), and D3 (third distance information) as the distance measurement value D. Distance measurement values ​​D1, D2, and D3 each have different relative and absolute errors depending on the type of distance measurement acquired. By selecting the distance measurement value that is thought to have small relative and absolute errors, the distance information generation unit 322 can adopt the distance measurement value with the smallest error from among multiple distance measurement methods, depending on the situation.

[0183] For example, the distance values ​​obtained by ground contact position distance measurement or imaging plane phase difference distance measurement have less error when the distance value is greater. Therefore, the distance information generation unit 322 may select either distance value D1 or distance value D2 if the acquired distance value is greater than (farther than) a predetermined distance, and select distance value D3 if it is less than (closer to) the predetermined distance. In addition, integrated distance information may be generated by weighting and adding at least two of the first to third distance information values.

[0184] Alternatively, as another integrated processing method, the absolute and relative errors can be considered for each, the probability distribution of existence for the distance can be calculated, and the sum of the probability distributions can be selected to maximize the probability of existence. Furthermore, based on the movement speed, accelerator, brake, and steering information of vehicle 100 included in data D1401, the probability of existence of the relative distance value can be determined for the current relative distance value. For example, to prevent large changes in acceleration, the probability can be determined such that the same acceleration as the previous time point has the highest probability, and the probability decreases as the acceleration change increases. The probability of existence of the relative distance can then be calculated accordingly.

[0185] Furthermore, if accelerator information is available, the maximum probability can be determined to increase acceleration, and if brake information is available, the maximum probability can be determined to decrease acceleration. In addition, this can be determined depending on the type of object being targeted. If the type (category) of the object being targeted is an automobile or motorcycle, the recognized object may accelerate or decelerate significantly, so the change in relative distance may be large.

[0186] On the other hand, if the category is pedestrians or similar and there is no sudden acceleration or deceleration, the change in relative distance is likely to depend on the user's own actions, allowing for a more accurate determination of the probability of existence. The above explains the basic flow when a single input data point at a specific time is received. Next, we will explain the case when a series of time-series input data points are received.

[0187] Figure 15 is a flowchart showing a portion of the processing example when multiple time-series consecutive input data are input in the second embodiment, and Figure 16 is a flowchart showing the remaining portion of the processing example shown in Figure 15. The computer included in the route generation device 150 executes a computer program stored in memory to perform each step of the flowcharts in Figures 15 and 16. The same processing and data as described above using Figure 14 are assigned the same reference numerals and their explanations are omitted.

[0188] When sequential input data is received in a time series, the distance measurement value D1 for D1402, the object width Ws for D1403, and the distance measurement value D2 for D1404 can be obtained sequentially in time series for objects that have been assigned the same identification number, i.e., objects that are recognized as the same object.

[0189] When continuous time-series input data is received, each distance measurement value changes over time due to the change in the relative distance to the target object. However, assuming the object is a rigid body, the actual object width W can be said to be always constant. Therefore, in step S1504, the distance information generation unit 322 smooths the actual object width W in the time-series direction even if the relative distance value changes over time, and obtains the average actual object width W' data D1505. This makes it possible to reduce relative errors.

[0190] When performing this analysis, it is advisable to remove outlier data to avoid using data with large errors. Furthermore, smoothing the time series across all data can leave initial errors in place over time. In such cases, this can be improved by taking a moving average within a predetermined range. The number of data points used for the moving average can be determined by considering both relative and absolute errors. For systems with large relative errors, a larger number of data points is preferable; conversely, for systems with large absolute errors, fewer points are better.

[0191] When W is assumed in equation 8, the relative error can be made sufficiently small by smoothing in the time series direction, including the ground contact position distance value and object width (number of pixels). However, an absolute error equivalent to the ground contact position distance value remains. The same consideration can be made when the actual object width W' is calculated using the image plane phase difference distance value. Data D1505 becomes the object width W' with small relative error, and through the distance conversion process in step S1405, the object width distance value D3 as data D1406 is obtained as a distance value where the absolute error is equivalent to the ground contact position distance value, with only the relative error of the object width Ws remaining.

[0192] On the other hand, in step S1607 of Figure 16, the distance information generation unit 322 calculates the correction amount for the absolute error of the image plane phase difference distance measurement value D2 based on the measured distance values ​​D1 and D3. The absolute error of the image plane phase difference distance measurement value D2 mainly consists of a component that is a constant value regardless of distance when converted to defocus. Therefore, the distance information generation unit 322 converts the image plane phase difference distance measurement value D2 into a defocus amount based on the focal length and imaging formula.

[0193] The distance information generation unit 322 similarly converts the object width measurement value D3 into a defocus amount using the same focal length and imaging formula. In this case, the ground contact position measurement value D1 or the relative measurement value D may also be used to convert to the defocus amount.

[0194] The distance information generation unit 322 calculates the average value of the time-series difference data by taking the difference between the defocus amount calculated from the image plane phase difference distance measurement value D2 and the defocus amount calculated from the object width distance measurement value D3 for data taken at the same time. If the average value can be calculated with sufficient data, the obtained average value will represent the difference in absolute error between the image plane phase difference distance measurement value D2 and the object width distance measurement value D3. If this average value is selected in step S1609, data D1608 will be used as the absolute error correction value.

[0195] Furthermore, when correcting using the grounding position distance value D1, the correction is performed using the absolute error of the grounding position distance value D1, and if the grounding position distance value D1 was used in step S1504, the result will be the same absolute error. In that case, selecting the one with the smaller relative error will reduce the influence of the relative error.

[0196] Step S1609 is the absolute error correction value selection process, in which the distance information generation unit 322 decides whether to select the result of step S1607 or the result of step S1608 as the absolute error correction value. Details will be described later.

[0197] In step S1402, the distance information generation unit 322 calculates the distance value D2 by image plane phase difference distance measurement and corrects the defocus amount using the absolute error correction value D1608. Since the absolute error correction value represents the defocus offset, by subtracting the offset from the defocus amount calculated from the input data, the distance value D2 in D1404 becomes one in which the absolute error has been corrected. Alternatively, the distance value may be corrected directly. In practice, this is done to match the absolute error of the data used for the difference in step S1607. As mentioned above, when using the defocus amount converted from the object width distance value D3, it becomes the absolute error of the object width distance value D3.

[0198] The object width distance value D3 depends on the distance value used in step S1504. Therefore, if the actual object width W was calculated using the ground contact position distance value D1 in step S1504, the ground contact position distance value D1, the image plane phase difference distance value D2, and the object width distance value D3 will all be the absolute error of the ground contact position distance value D1. Since the absolute errors are the same for all three distance values, in the integrated distance measurement process in step S1406, only the relative error can be considered when determining the probability distribution. This makes it possible to calculate the relative distance value D of data D1507 more simply and stably.

[0199] As described above, by inputting time-series data of an object, a relative distance value D can be calculated in time series from the ground contact position distance value D1, the image plane phase difference distance value D2, and the object width distance value D3. From the time-series data of the relative distance value D, the relative velocity, relative acceleration, and relative jerk with respect to the target object can be calculated. Using these, the probability distribution of the relative distance value mentioned above can be calculated. For example, the probability distribution can be determined so that the change in relative acceleration is small. Furthermore, the calculation of the distance value D can also be modeled, including the variation in relative distance values. In that case, if it can be expressed as a linear combination of each distance value, it would look like equation 10.

[0200]

number

[0201] In other words, if we can estimate K1, K2, and K3 from the probability distributions of each method, we can obtain a plausible distance measurement value D. By assuming that the probability distributions of each distance measurement value Dm, D1, D2, and D3 can be approximated by a Gaussian distribution, we can solve it analytically using equation 11. Specifically, by taking the variance of both sides and taking the partial derivative with respect to the coefficient K, we can obtain three equations for the three unknowns.

[0202]

number

[0203] Similarly, the object width distance measurement value D3 depends on the recognition accuracy on the image that determines the object width. Regarding the image plane phase-difference distance measurement value D2, when calculated using general block matching, there are many distance measurement points on the recognized image, and the variance can be calculated as a statistical measure. However, this can include large errors, so it is better to remove outliers from the block matching results. Also, since it is obtained as a statistical measure of many distance measurement values ​​within the recognized object, it depends on the number of distance measurement points, and it is necessary to consider the block size of the block matching in this case.

[0204] This is because, when considering each distance measurement point, the blocks of block matching overlap in neighboring distance measurement points, and therefore the noise influence of the sensor pixels is not independent. If the variance of the average phase difference distance measurement value D2 within the recognized object were independent of the variance of each distance measurement point, it would be the variance of each distance measurement point divided by the number of distance measurement points. However, as mentioned above, since it is not independent for each distance measurement point, it is necessary to take this into account and determine a worsened value for the variance. Specifically, this can be done by calculating the product of the number of pixels within the block, or by multiplying it by a value determined by some other method.

[0205] Furthermore, Vm can be updated using the results of the previous frame. Specifically, the posterior probability in the above model can be calculated, and the Vm can be updated by combining it with the posterior probability (mainly by adding it to the assumed variance). In this example, formulas for three distance values ​​have been provided, but if there are other distance measurement methods, these can be increased, and in that case as well, the analytical solution can be obtained using a similar approach to formula derivation.

[0206] Conversely, it's not necessary to use all three D1, D2, and D3. You can use only the ones you select, depending on the situation. In particular, when calculating changes in velocity, acceleration, etc., if the amount of variation is large, a large error can be expected, so in that case, you might consider not using them.

[0207] For example, if it is known that the object ahead is stationary, the approximate change can be calculated from the vehicle's speed. Also, the variance of the distance measurement value D (the probability distribution assuming a Gaussian distribution) can be calculated using equation 11, so if the distance measurement values ​​D1, D2, and D3 deviate significantly from these, it is highly likely that there is an error in the distance measurement method. In that case, the coefficient K can be set to 0 to disregard it.

[0208] We have discussed time-series data so far, but it is not always necessary to acquire the data continuously. If object recognition fails and correct input data is not obtained for that frame, processing will not be performed, and recalculation can be started from that point when recognition is successful.

[0209] In that case, the average value W' of the actual object width and the absolute error correction value can be used as is, and a stable relative distance value D can be obtained from them. However, in that case, the probability distribution of the relative distance value using the relative velocity, relative acceleration, and relative jerk calculated from the relative distance value D cannot be correctly determined, so it is desirable to set it to 0.

[0210] Furthermore, there may be cases where one or two of the following values ​​cannot be calculated correctly: the ground contact position distance value D1, the image plane phase difference distance value D2, and the object width distance value D3. For example, this could occur if the recognized object is floating, or if another object is positioned between the recognized object and the object width, preventing accurate measurement of the object width. In such cases, the change may differ significantly from the previous value or significantly from other distance values, making it highly likely that the value cannot be calculated correctly. In such cases, the influence on the calculation of the relative distance value D can be avoided or mitigated by setting the probability distribution of that distance value to 0 or a very small value.

[0211] In step S15108, the distance information generation unit 322 performs a correction process that uses the change in relative distance and the change in the width W of the actual object to perform a highly accurate absolute error correction when the relative distance value D with respect to the target object has changed sufficiently as a time series change.

[0212] Figures 17(A) and (B) are schematic diagrams illustrating an example of absolute error correction processing in the second embodiment. The horizontal axis represents time, with the distance value D plotted on the vertical axis in Figure 17(A) and the actual object width W plotted on the vertical axis in Figure 17(B). Figure 17(B) shows the actual object width W calculated from the object width at each time point, not the average actual object width W'. Relative errors have been removed for illustrative purposes.

[0213] Even when the relative distance value D changes as shown in Figure 17(A), the actual object width W should remain constant as shown by the dotted line in Figure 17(B). However, if absolute error is included, it will change depending on the distance, as shown by the solid line. Using the true distance value Dt, the true actual object width Wt, and the estimated average actual object width We', the estimated distance value De with error can be expressed by the following equation (Equation 12).

[0214]

number

[0215] Therefore, in step S11608, an absolute distance correction component is determined so that, if a certain relative distance fluctuation occurs, no change in the actual object width occurs at each time point. Since the true distance value Dt is unknown, it can be estimated from the fluctuation component and the change in the actual object width W by considering one or more of the ground contact position distance value D1, the imaging plane phase difference distance value D2, and the relative distance value D.

[0216] Specifically, for example, the actual object width W is calculated using the image plane phase difference distance value D2 in a method equivalent to step S1404, and the change in that time series is calculated. The absolute error correction value should be adjusted so that the change in the actual object width W is minimized. A general optimization method can be used to find the correction value that minimizes the change in the actual object width W. This means that the absolute error correction value and the actual object width correction value are estimated simultaneously.

[0217] Let me explain one specific calculation method. The imaging formula, which is the relationship between the measured distance value and defocus in image plane phase-difference distance measurement, and the relationship between the measured distance value and the image plane in object width distance measurement, can be rearranged as follows, assuming that errors are included and the order of the distance value is sufficiently larger than the order of the defocus, to obtain the following equation 13.

[0218]

number

[0219] In other words, as the relative distance value D approaches 0, the error in Wt approaches 0. Therefore, by plotting the relative distance value D against the reciprocal of the actual vehicle width, 1 / Wt, the intercept becomes the correct value of the actual vehicle width, from which the absolute error correction value can be calculated. In reality, since relative errors are also included in addition to absolute errors, the data plot will vary, but if there is a certain relative distance variation and a certain amount of data for a certain time period, it can be easily solved using methods such as the least squares method.

[0220] Furthermore, the absolute errors of the corrected image plane phase difference distance measurement value D2, object width distance measurement value D3, and ground contact position distance measurement value D1 should be the same, and the difference in these absolute errors can also be calculated as the absolute error correction amount. In the ground contact position distance measurement process of step S1401, the pitch angle deviation of the imaging device mentioned above is the main cause of the absolute error component. In other words, the vanishing point is at a different position than expected, and this component becomes the absolute error of the ground contact position distance measurement value.

[0221] In step S1608, the absolute error component can be estimated, allowing for the estimation of the pitch angle deviation. This correction amount can then be defined as the absolute error correction amount (data D1610). When relative distance fluctuations occur in this way, this information can be used to correct the absolute error component of each distance measurement with greater accuracy.

[0222] If the fluctuations of the absolute error component, including its time series, are negligible, the process can proceed from the overall absolute error correction process in step S1608 to the actual object width acquisition process in step S1504. In other words, the calculation of the average value of the actual object width and the absolute error correction process in step S1607 are basically unnecessary. Therefore, each of these processes can be simplified, or they can be continued for verification purposes.

[0223] In step S1609, the distance information generation unit 322 selects whether to output the absolute error correction value from step S1607 or step S1608 as data D1608. As mentioned above, step S1608 is performed when a certain relative distance fluctuation occurs.

[0224] Therefore, in step S1609, the distance information generation unit 322 basically selects the absolute error correction value calculated in step S1607, and if the processing in S1608 is performed, it selects the absolute error correction value calculated in step S1608.

[0225] If there is a change in the absolute error, the absolute error correction value calculated in step S1607 may be selected again. For example, if the absolute error correction value calculated in step S1608 changes after the absolute error correction value calculated in step S1607 has been selected, the absolute error correction value calculated in step S1607 can be selected again. Subsequently, if a certain relative distance change occurs again, step S1608 is performed and that absolute error correction value is selected, and in this way, a better absolute error correction value can be continuously selected.

[0226] As described above, Figures 15 to 17 illustrate the flow for correcting relative and absolute errors over time using ground position distance measurement, object width distance measurement, and image plane phase difference distance measurement. Although the above explanation focused on image plane phase difference distance measurement, the same approach can be applied to stereo distance measurement. Relative error is equivalent to the effect of sensor noise, and absolute error can be considered to be the effect of the installation position error of each stereo imaging device. While absolute error is not converted into a defocus value, the correction amount can be estimated as an installation position error.

[0227] Furthermore, the same approach can be applied to other modalities such as LiDAR. For example, with LiDAR, relative errors in distance resolution occur, and absolute errors also occur as distance offsets, so the same approach as in this embodiment can be applied. In this example, we have described an integrated flow of three distance measurement methods, but it is also possible to use an integrated flow with two methods extracted from each. In addition, by adding image plane phase-difference distance measurement, stereo, and LiDAR (other modalities), it is possible to correct for four or more combinations of distance measurement methods in the same way.

[0228] This integrated distance measurement method allows for stable distance measurements for purposes such as tracking a recognized object for a set period of time. By installing this imaging device on a vehicle, it can be applied to applications such as ACC (Adaptive Cruise Control) and autonomous driving.

[0229] Thus, in this embodiment, by using image information in addition to information from the usual modality, it is possible to obtain object distance information of the object in question with higher accuracy. Furthermore, in this embodiment, the route generation device 150 generates route information based on the integrated distance information generated in this way. Therefore, it is possible to generate route information with higher accuracy. Here, the route generation device 150 functions as a route generation means that executes a route generation step of generating route information based on integrated distance information. The route generation device 150 generates route information based on the speed of the vehicle as a moving object.

[0230] <Third Embodiment> The path generation device 150 of the third embodiment achieves high-precision distance measurement from short to long distances by combining distance measurement using parallax with distance estimation based on a single image.

[0231] Figure 18 is a block diagram showing an example configuration of a distance measuring system according to a third embodiment. In this embodiment, the distance measuring system is included in the imaging device 110. The distance measuring system comprises an image sensor 1802 as an image acquisition unit, a recognition processing unit 1821, a distance image generation unit 1812, a scaling distance measuring unit 1803, and a distance correction unit 1804. Here, the image sensor 1802 has the same configuration as the image sensor 302 in the first embodiment. The recognition processing unit 1821 corresponds to the recognition processing unit 321. The distance image generation unit 1812 corresponds to the distance image generation unit 312. The scaling distance measuring unit 1803 and the distance correction unit 1804 correspond to the distance information generation unit 322.

[0232] The image sensor 1802 acquires image signals in the same way as the image sensor 302. The image sensor 1802 has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412 arranged within each pixel. It also acquires an image signal composed of the image signal acquired by the first photoelectric conversion unit 411 and an image signal composed of the image signal acquired by the second photoelectric conversion unit 412.

[0233] These are images corresponding to different viewpoints and are called disparity images. The image sensor 1802 acquires a composite image signal obtained by combining the image signals of the two disparity images as the captured image. Alternatively, the image sensor 1802 may acquire one of the two disparity images as the captured image.

[0234] Furthermore, the camera configuration for obtaining the parallax image may use stereo cameras arranged side by side. Alternatively, a monocular camera configuration may be used, and the parallax image may be obtained by considering the speed of the vehicle and treating the relative movement of objects in consecutive frame images as parallax.

[0235] The recognition processing unit 1821 applies image recognition processing to the captured image captured by the imaging device 1802 to detect an object included in the captured image. In order to realize the automatic driving control and collision mitigation brake control of the vehicle 100, it is necessary to recognize an object such as a lane in which the vehicle 100 travels, a vehicle (preceding vehicle) traveling in front of the vehicle 100, or a person. As a method for object detection, a method based on template matching for detecting an object whose appearance is almost constant (such as a signal or a traffic sign), or a method for detecting a general object (such as a vehicle or a person) using machine learning is used.

[0236] In the present embodiment, the recognition processing unit 1821 executes a lane line detection task and an object recognition task. FIG. 19 is a schematic diagram showing an example of output results of the lane line detection task and the object recognition task executed by the recognition processing unit 1821 according to the third embodiment. The lane line detection task uses a machine learning model that takes a captured image as input and detects whether or not each pixel is a lane line (or a white line or a yellow line) on the road, and labels whether or not it is a lane line (in the figure, the detected lane line is indicated by a black broken line). ) to obtain a lane line region map.

[0237] The object recognition task takes a captured image as input and uses a machine learning model that detects an object on the road to obtain the type of the detected object (person, vehicle, sign, etc.), the coordinates (x0, y0) of the upper left point of the detection frame, and the coordinates of the lower right point of the detection frame (x1, y1). Then, the coordinates of the detection frame in contact with the detected object are obtained. Here, it is assumed that the output of the object recognition task is equivalent to the external information shown in the first embodiment. [[ID=?]]<00009?0>[[ID=?]]<00009?1>The distance image generation unit 1812 obtains distance data from the disparity image obtained by the imaging device 1802. Distance measurement from the disparity image can calculate the disparity value by detecting corresponding points between images with different viewpoints, and calculate the distance from the disparity value and the camera conditions (focal length, baseline length) for capturing the disparity image.

[0239] As described above, even if the camera that calculates the parallax image is a monocular camera using a dual-pixel CMOS sensor, it is possible to specify the camera conditions for distance calculation. Generally, in distance measurement using a parallax image, it is known that the distance estimation accuracy deteriorates because the parallax almost disappears as the distance measurement target moves farther away.

[0240] The scaling distance measurement unit 1803 calculates the distance value of the second region by scaling the distance value of the first region calculated by the distance image generation unit 1812 according to the size ratio between the object existing in the first region and the object existing in the second region. In the present embodiment, an example will be described in which the distance value on the near-distance side calculated by the distance image generation unit 1812 is expanded to the far-distance side for the road surface on the near-distance side, and scaling is performed based on the road information on the near-distance side and the far-distance side.

[0241] FIG. 20 is a diagram for explaining an example of the positional relationship between the camera mounting position and the road surface. A global coordinate system (X, Y, Z) is set with the road surface in the vertical direction of the camera mounting position V(0, Y0, 0) as the origin O. Also, the imaging optical system 301 of the imaging camera used for the imaging element 1802 and the optical axis of the imaging element are assumed to be arranged such that the optical axis direction is horizontal. Also, the imaging range is shown as the range to be imaged according to the viewing angle of the imaging camera. If the image width of the imaging range, that is, the imaging image and the distance map calculated by the distance image generation unit 1812 is wu, and the image height is hv, the center pixel of the distance map is expressed as c(wu / 2, hv / 2).

[0242] First, the road surface on the near-distance side is estimated. The area assumed to be the road surface on the near-distance side on the distance map is the area at the lower part of the distance map. Alternatively, it is also possible to directly estimate the road surface on the near-distance side by the recognition processing of the captured image. Among the areas at the lower part of the distance map, the periphery of the pixel determined to be a dividing line on the dividing line area map detected by the recognition processing unit 1821 may be regarded as the road surface. Only the distance map area determined to be closer than the threshold value determined according to the camera installation conditions (viewing angle, resolution, line-of-sight angle), shooting environment (weather, time zone), and information on the road on which the vehicle travels (lane width, number of lanes, branch / merge points, road type) may be regarded as the road surface.

[0243] If we let D be the distance value of pixel p(u0,v0) on the distance map, then the nearby road surface p(u0,v0) can be transformed into global coordinates X=u0-wu / 2, Y=v0-hv / 2, Z=D. Assuming the road surface is horizontal, the equation of the surface is expressed as aX + bY + cZ + d = 0 (where a, b, c, and d are constants). The road surface can be estimated by determining the constants in the above equation using four or more points that represent the road surface on the near side.

[0244] The estimated road surface equation makes it possible to estimate the road surface on the far side. Furthermore, the distance can be scaled from this extended far-side road surface. Specifically, the depth Z of point R(X,Y,Z) located at the intersection of the road surface equation and a line passing through the viewpoint V(0,Y0,0) and pixel q(u1,v1) which represents the far-side region on the distance map can be estimated as a distance value.

[0245] Even if the scaling distance measurement unit 1803 cannot obtain the road distance corresponding to pixel q(u1,v1) from the distance data, it is possible to obtain the road distance value Z corresponding to q(u1,v1) by performing the calculation as described above. If we consider this process on a distance map, the distance in the first region is scaled by the ratio of road surfaces in the near-field region (first region) to road surfaces in the far-field region (depth ratio in 3D space) to calculate the distance in the second region. The distance values ​​calculated in this way do not use the distance values ​​from the parallax distance measuring unit, which deteriorates at long distances, thus improving the accuracy of distance estimation at long distances.

[0246] Furthermore, the distance measurement accuracy can be improved by using the detection frame detected by the recognition processing unit 1821. Figure 21 is a schematic diagram showing an example of a scene in which two signs of known size are detected on a road. Assume that in the acquired image, one sign of known size is placed on the near side and one on the far side. In this state, the distance to the near-field sign can be accurately calculated by the distance image generation unit 1812. Also, if the object size is known, the scaling distance measurement unit 1803 can calculate the distance to the far-field sign based on the ratio of the number of pixels in the image.

[0247] Let w0 be the height (number of pixels) of the sign at the near distance, d0 be the distance from the camera position, and w1 be the height (number of pixels) to the sign at the far distance, and d1 be the distance from the camera position. The scaling distance measuring unit 1803 can calculate the distance using the formula d1 = d0 * (w0 / w1). As described above, if objects of known size (such as signs and traffic signals) or objects that can be assumed to have the same size at the near and far distances (such as guardrails, width, length, and spacing of road markings) can be detected, high-precision scaling distance measurement can be performed.

[0248] When performing scaling distance measurement with an object of known size, it is difficult to improve accuracy across the entire distance range, from near to far. However, by combining the scaling distance measurement that extends the road surface as described above with the scaling distance measurement that utilizes the object size ratio, it becomes possible to perform highly accurate scaling distance measurement.

[0249] The distance correction unit 1804 corrects the distance value measured by the distance image generation unit 1812 based on the distance value calculated by the scaling distance measurement unit 1803 to obtain a corrected distance value Dc. Hereinafter, the value measured by the distance image generation unit 1812 will be called the distance value D, and the distance value calculated by the scaling distance measurement unit 1803 will be called the scaling distance measurement value Ds. The corrected distance value Dc is calculated according to the following equation (Equation 14), with coefficient α.

[0250]

number

[0251] (1) Method for determining the coefficient α based on the magnitude of the distance value Regarding the distance measurement by the parallax distance measuring unit, the distance value to the object being measured itself affects the measurement accuracy. As the distance to the object increases, the distance accuracy of the distance value calculated by the parallax distance measuring unit decreases. Therefore, the coefficient α is determined so that the proportion of the scaled distance measurement value increases according to the distance value. This allows for obtaining accurate measurement results. In other words, the coefficient α is determined so that the larger the distance value D, the smaller α becomes.

[0252] (2) Method for determining the coefficient α based on the contrast of the distance measurement target Regarding the distance measurement of the parallax distance measuring unit, in addition to the distance value to the object being measured, the contrast of the object is another factor that affects the measurement accuracy. When calculating the corresponding points (parallax) between parallax images, even if low-contrast areas are matched, the distinction from surrounding areas is unclear, making it impossible to accurately determine the parallax.

[0253] Therefore, if the target to be measured is not sufficiently illuminated at night and its contrast is low, the system determines that the accuracy of the distance measurement is low and sets the coefficient α to increase the proportion of the scaled distance measurement value. In other words, the coefficient α is determined to be large when the contrast of the target to be measured is low. Since the scaled distance measurement value is generated based on the highly accurate distance value of the area illuminated by, for example, vehicle lighting, accurate distance measurement results can be obtained even when the contrast of the target to be measured is low.

[0254] (3) Method for determining coefficients based on the category type of the detection frame Depending on the type (category) of object being measured, the distance measurement accuracy tends to differ between parallax distance measurement and scaling distance measurement. For example, for objects that are far from the scaling reference, such as the light-emitting part of a traffic light, parallax distance measurement can measure the distance without distinguishing it from other objects, but scaling distance measurement tends to have lower accuracy.

[0255] Therefore, when the category of the detection frame is a specific type, by setting the coefficient so that the ratio of the distance value increases, an accurate distance measurement result can be obtained. Also, as shown in FIG. 21, when the distance accuracy of the scaling distance measurement in the peripheral area where the object is detected increases by the method of scaling distance measurement, the coefficient may be determined so that the ratio of the scaling distance measurement around the detection frame increases according to this category. The coefficient determination method shown above does not need to be limited to one, and the final coefficient may be determined based on the coefficients generated for each factor. As described above, an accurate distance measurement result can be obtained.

[0256] (Modified example of the scaling distance measurement unit) As a modified example of the scaling distance measurement unit 1803, an example of estimating the distance to the target object by scaling the distance data in the vicinity based on the size ratio of the road width on the images on the near side and the far side will be described. FIG. 22 is a block diagram showing the configuration of a modified example of the scaling distance measurement unit 1803.

[0257] The scaling distance measurement unit 1803 includes a lane analysis unit 2201, a roll angle estimation unit 2202, a ground contact position estimation unit 2203, and an object distance calculation unit 2204. The lane analysis unit 2201 detects the number of pixels between the lane lines as the lane width from the lane line area map, and detects the center coordinates between the lane lines as the lane center (coordinates).

[0258] FIGS. 23(A) to (D) are schematic diagrams schematically showing processing examples of lane width detection and lane center detection executed by the lane analysis unit 2201 according to the third embodiment. FIG. 23(A) is a diagram schematically showing a lane line area map showing lane lines in an image. Pixel detected as a lane line in the image by the lane analysis unit 2201 is given a lane line flag (represented by a black broken line in the figure). Since the lane lines on the road are shown by broken lines at the boundary between the driving lane and the overtaking lane, or the lane lines are hidden by vehicles or obstacles on the road, the lane line area map appears intermittently.

[0259] Figure 23(B) is a schematic diagram illustrating a method for calculating lane width and lane center position. The lane analysis unit 2201 checks each pixel in the lane marking area map from left to right to determine whether a lane marking flag is set. If the pixel being checked does not have a lane marking flag, and the pixel to its left has a lane marking flag, that pixel is considered the starting point of the road width. Similarly, if the pixel being checked has a lane marking flag, and the pixel to its left does not have a lane marking flag, that pixel is considered the ending point of the road width. As a result, as shown in the group of arrows in Figure 23(B), the lane width (length of the arrow) and the lane position (center position of the arrow) can be detected one by one for each line of the lane marking area map.

[0260] Figure 23(C) is a schematic diagram showing the detected lane width data. Lane width is observed with a number of pixels proportional to the reciprocal of the distance from the camera, as long as the actual size of the lane does not change. In Figure 23(C), the horizontal axis shows the lines of the lane marking area map, and the vertical axis shows the detected lane width (number of pixels). As shown, lane width has a highly linear relationship with the lines. Depending on the degree of breaks in the lane markings, lane widths for one lane and lane widths for two lanes may be observed together, but since the difference is approximately twice as large, it is easy to separate them.

[0261] Figure 23(D) shows a schematic diagram illustrating the interpolation of observed data for the separated left lane and data for lines where lane width could not be obtained from the observed data. Regarding the separation of adjacent lanes (left and right lanes), it is easy to exclude observed data for adjacent lanes by comparing the position of the lanes (center position of the arrows). From this information, the lane width for only the left lane shown in Figure 23(B) can be obtained using robust estimation methods such as the RANSAC method.

[0262] For lane width interpolation, the equation of an approximate straight line can be calculated using the RANSAC method described above, and this approximate straight line equation can be used, or interpolation can be performed by interpolating between observed road width data. The position of the lanes can also be obtained for each line using the same method as described above. Using the lane information obtained above (lane width and lane center), scaling distance measurement is performed.

[0263] The roll angle estimation unit 2202 functions as a roll angle estimation means that estimates the roll angle of the camera from the distance map. There are several factors that cause a roll angle for an in-vehicle camera. For example, unevenness in the road surface can cause a difference in height between the contact surfaces of the left and right tires, which can prevent the camera mounting position from remaining horizontal and cause a roll angle. Also, when a vehicle turns a curve, centrifugal force can deform the vehicle body itself, causing a roll angle. Such roll angles significantly affect distance estimation.

[0264] The scaling distance measurement described here is a process that scales the distance to an object on the road surface using the ratio of the number of lane width pixels assumed to be at the same distance as the object to the number of neighboring lane width pixels, thereby scaling the distance to the neighboring object with high accuracy. When the camera rolls, it becomes difficult to find the lane width assumed to be at the same distance as the object.

[0265] Figure 24 is a schematic diagram comparing the position of lane width in captured images with and without roll angle. The image on the left shows the case without roll angle, while the image on the right shows the same scene with a roll angle of 10° centered on the lower left of the image. In the image on the left, the ground contact position of the vehicle being measured and the lane assumed to be at the same distance are aligned on the same straight line, making correspondence easy. On the other hand, in the image on the right where a roll angle is present, there is no lane on the same line as the ground contact position of the vehicle being measured, and even if there were a lane, it would be impossible to calculate the lane width at the same distance. To find the lane width at the same distance as the vehicle being measured, it is necessary to estimate the roll angle accurately.

[0266] The roll angle estimation unit 2202 estimates the roll angle based on the distance map obtained by parallax distance measurement. When a roll angle occurs, it is known that the position at which the distance to the road surface, which can be assumed to be horizontal, is the same, is tilted, as shown in Figure 24. Since the roll angle is a rotation around the viewpoint direction, there is no change in the distance between the camera and the target object, and the image captured by the camera is tilted due to the roll angle, so the distance map also tilts in line with the tilt of the road surface in the image.

[0267] Figure 25 is a flowchart showing an example of the roll angle estimation process performed by the roll angle estimation unit 2202 according to the third embodiment. Figures 26(A) to (C) are schematic diagrams illustrating each processing example in the roll angle estimation process according to the third embodiment. Note that the computer included in the path generation device 150 executes a computer program stored in memory to perform each step of the flowchart in Figure 25.

[0268] In step S2500, the roll angle estimation unit 2202 determines the target pixel setting range from the distance map. Figure 26(A) is a schematic diagram showing the target pixel setting range in the distance map.

[0269] As shown in Figure 26(A), the area to be set to the pixel of interest is located in the lower left region of the distance map. The height of the area to be set to the pixel of interest is assumed to correspond to the road surface, for example, from the horizontal line to the bottom edge of the distance map. The width of the area to be set to the pixel of interest is set taking into account the distance between the pixel of interest and the search area. The search area is set at a position a predetermined number of pixels (a predetermined interval) away horizontally from the pixel of interest.

[0270] If the distance between the pixel of interest and the search area is large, even if the width of the pixel of interest setting range is set large, it becomes impossible to set the search area. In this embodiment, the predetermined distance is set to about 1 / 4 of the image width of the distance map, and the size of the pixel of interest setting range is set to about half of the image width of the distance map. By appropriately setting the pixel of interest setting range, the amount of computation can be reduced.

[0271] In step S2501, the roll angle estimation unit 2202 obtains distance data for the target pixel 2600 from the distance map. In this embodiment, distance data is obtained for the target pixel 2600 within the target pixel setting range.

[0272] In step S2502, the roll angle estimation unit 2202 determines the search range. Regardless of whether a roll angle occurs or not, the distance values ​​of pixels near the pixel of interest tend to be close to the distance value of the pixel of interest. Therefore, in order to detect the roll angle with high resolution from the coordinates of the pixel of interest 2600 and the corresponding pixel 2601, the roll angle estimation unit 2202 sets a search range that is spaced horizontally at a predetermined interval from the pixel of interest 2600.

[0273] The roll angle estimation unit 2202 can limit the height of its search range by the expected roll angle range. For example, in the case of an in-vehicle camera installed in a vehicle, when driving on a typical road, the roll angle is limited to ± a few degrees. In this embodiment, it is set to about 1 / 8 of the image height of the distance map.

[0274] The search range width is set from a position a predetermined distance horizontally from the target pixel 2600 to the right edge of the distance map. The right edge of the search range is not limited to the right edge of the distance map, but if the search range width is too small, the area may be too small to find the distance data for the road surface corresponding to the target pixel 2600. Therefore, it is preferable to set the search range width as large as possible.

[0275] In step S2503, the roll angle estimation unit 2202 searches for a corresponding pixel 2601 that corresponds to the target pixel 2600 within the search range. To position a pixel similar in distance value to the target pixel 2600 within the search range as the corresponding pixel 2601, the difference between the distance value of the target pixel 2600 and each pixel within the search range is detected, and the pixel with the smallest difference is designated as the corresponding pixel 2601.

[0276] Furthermore, the method for finding the corresponding pixel 2601 is not limited to comparing the difference between one pixel and another. Alternatively, the difference in distance values ​​between neighboring pixel groups, including the pixel of interest and the pixels within the search range, may be compared, and the central pixel of the pixel group with the highest similarity may be designated as the corresponding pixel 2601.

[0277] In step S2504, the roll angle estimation unit 2202 calculates the inclination θ from the coordinates of the pixel of interest and the corresponding pixel. As shown in Figure 26(B), if the coordinates of the pixel of interest 3800 are (x0, y0) and the coordinates of the corresponding pixel 2601 are (x1, y1), the inclination θ is calculated by θ = arctan((y1-y0) / (x1-x0)).

[0278] In step S2505, the roll angle estimation unit 2202 branches depending on whether it has completed processing all pixels within the target pixel setting range. If it has completed processing all pixels within the target pixel setting range as target pixels, it proceeds to step S2506. If it has not completed processing all pixels within the target pixel setting range as target pixels, it proceeds to step S2501 and processes new target pixels.

[0279] In step S2506, the roll angle estimation unit 2202 calculates the roll angle. The flow shown in Figure 25 is repeated periodically until the user issues a termination instruction (not shown).

[0280] Figure 26(C) is a schematic diagram showing the distance value of the pixel of interest on the horizontal axis and the slope calculated for that pixel on the vertical axis. Since the slope calculated from a single pixel of interest contains noise components, the most plausible roll angle is detected by averaging multiple slopes. The distance map used for roll angle estimation is generated from the disparity image, but it is known that the detection accuracy of distance measurement from the disparity image deteriorates as the distance to the object increases.

[0281] Therefore, when calculating the slope for each pixel of interest using a distance map, the variability of the calculated slope increases with the magnitude of the distance value of the pixel of interest. For this reason, in calculating the roll angle, the roll angle is estimated by a weighted average that increases the proportion of slopes with small distance values ​​and decreases the proportion of slopes with large distance values. Alternatively, the similarity score used during the corresponding pixel search may be used as a factor in determining the weighted average. Through the above process, the roll angle can be estimated using a distance map of the road surface.

[0282] Furthermore, we will explain how to set the interval between the pixel of interest and the search range when the resolution of the roll angle to be estimated is predetermined. The distance between the pixel of interest and the search range is determined by the roll angle resolution. The roll angle is calculated as the slope from the pixel of interest to the corresponding pixel, and is expressed as the ratio of the horizontal difference between the pixel of interest and the corresponding pixel to the vertical difference between the pixel of interest and the corresponding pixel. Since the vertical difference is at least 1 pixel, the roll angle resolution is determined by the magnitude of the horizontal difference. The distance d between the pixel of interest and the search range is obtained using the roll angle resolution r and the following equation (Equation 15).

[0283]

number

[0284] The larger the calculated interval d, the higher the resolution. However, as mentioned earlier, increasing the interval d narrows the search range, so it is preferable to set a minimum resolution. For example, if the required resolution r is 0.1°, the interval d will be 573 pixels or more.

[0285] The above calculation method allows for setting an appropriate interval when the roll angle detection resolution is given.

[0286] The ground contact position estimation unit 2203 uses the detection frame obtained by the object detection task, the lane center obtained by the lane analysis unit 2201, and the roll angle estimated by the roll angle estimation unit 2202 to estimate the coordinates of the lane width data located at the same distance as the ground contact position of the distance measurement target. Figure 27 is a flowchart showing an example of the lane width data coordinate estimation process performed by the ground contact position estimation unit 2203 according to the third embodiment. Note that the computer included in the route generation device 150 executes a computer program stored in memory to perform each step of the flowchart in Figure 27.

[0287] In step S2701, the ground position estimation unit 2203 selects a detection frame for distance measurement from the detection frames obtained in the object recognition task. In step S2702, the grounding position estimation unit 2203 sets the center position of the lower part of the detection frame as the coordinates (xc,yc) of the grounding position of the distance measurement target.

[0288] Figure 28 is a schematic diagram showing an example of the coordinates (xc, yc) of the grounding position of the distance measurement target set by the grounding position estimation unit 2203 according to the third embodiment. The coordinates (xc, yc) of the grounding position of the distance measurement target are expressed using the upper left coordinate (x0, y0) and lower right coordinate (x1, y1) of the detection frame, using the following equation (Equation 16).

[0289]

number

[0290] Furthermore, a method other than the one described above may be used to obtain the coordinates of the ground contact position of the object to be measured. With wide objects such as cars, the object will appear tilted by the roll angle, which can lead to a large discrepancy between the center coordinate of the lower edge of the detection frame and the coordinates where the tire is actually in contact with the ground. In such cases, for example, the contour of the object within the detection frame may be detected using the captured image or a distance map, and the position where a straight line R with a roll angle inclination touches the lower edge of the object's contour may be set as the coordinates of the ground contact surface. By using such a configuration, it is possible to set the coordinates where the object to be measured is actually in contact with the ground.

[0291] Since the process of recognizing the contour of an object is computationally intensive, a configuration that takes computational cost into consideration may be used to select an appropriate coordinate from the corners of the detection frame depending on the positional relationship between the lane center and the object to be measured. Specifically, one selection method is to select the coordinate of the lower left of the detection frame if the object to be measured is to the left of the lane center, and the coordinate of the lower right of the detection frame if it is to the right. This may allow the straight line R, which will be determined in the next step, to be set to a coordinate where the object is actually in contact with the ground, which is closer than setting the center of the lower edge of the detection frame.

[0292] In step S2703, the ground contact position estimation unit 2203 determines the coordinates (xt, yt) where a straight line R passing through the ground contact position coordinates (xc, yc) of the distance measurement target intersects with the straight line C of the lane center obtained in the lane mark detection task. By processing as described above, even if the distance map is tilted due to the roll angle, it is possible to determine the coordinates of the lane width data that are at the same distance from the ground contact point of the distance measurement target.

[0293] The object distance calculation unit 2204 calculates the distance to the distance measurement target using the coordinates on the lane width data corresponding to the ground contact position obtained by the ground contact position estimation unit 2203, the lane width data calculated by the lane analysis unit 2201, and the data on the distance map. Figure 29 is a flowchart showing an example of the distance estimation process to the distance measurement target performed by the object distance calculation unit 2204 according to the third embodiment. Note that the operation of each step in the flowchart of Figure 29 is performed by the computer included in the route generation device 150 executing a computer program stored in memory.

[0294] In step S2901, the object distance calculation unit 2204 calculates the lane width N2 using the lane width data and the coordinates (xt, yt). The object distance calculation unit 2204 may obtain the lane width at coordinates (xt, yt) as the lane width N2, or it may obtain multiple lane widths near coordinates (xt, yt), weight them according to the distance from coordinates (xt, yt), and obtain the N2 by weighting them and taking a weighted average.

[0295] When determining lane width from lane lines detected by image recognition, there is a possibility of discrepancies between the detected lane width and the actual lane width due to errors in image recognition. As mentioned above, calculating the lane width N2 using multiple lane widths can reduce the discrepancy in the detected lane width. Depending on the processing load, the number of lane width data used for smoothing may be limited to one.

[0296] In step S2902, the object distance calculation unit 2204 associates and saves the lane width data and distance data obtained in the current frame. First, the object distance calculation unit 2204 obtains each lane width data and its coordinates, and obtains distance data corresponding to the coordinates of the lane width data from the distance map. At this time, the corresponding distance data may be obtained by obtaining distance data corresponding to each coordinate within the range of the lane width data (xn-N1 / 2, yn) to (xn+N1 / 2, yn) centered on the coordinate (xn, yn) corresponding to the lane width N1 of the lane center line C, and then smoothing the data.

[0297] The object distance calculation unit 2204 performs this correspondence between lane width data and distance data for each lane width data obtained in the current frame T0. Figure 30 is a schematic diagram showing an example of each lane width data according to the third embodiment. The object distance calculation unit 2204 may narrow the range of data for which it performs correspondence depending on the processing load and recording capacity, and may be configured to save only the data for locations where the accuracy of the distance data is high.

[0298] In step S2903, the object distance calculation unit 2204 calculates reference data B to be used to calculate the distance to the object to be measured. When there are k data points associated in the previous step, the reference data B is expressed by the following equation (Equation 17) using the distance to the lane D1 and the lane width N1.

[0299]

number

[0300] In equation 16, D1[n]×N1[n] is the value corresponding to the lane width used as the basis for distance calculation. If D is the distance to the lane, N is the number of pixels representing the lane width in the image, F is the horizontal field of view of the imaging camera, and H is the horizontal image width, then the actual lane width W can be calculated by W=(D×N)×2×tan(F / 2) / W. The 2×tan(F / 2) / W part can be treated as a fixed value once the specifications of the imaging camera are determined.

[0301] Therefore, the actual lane width becomes a variable of the order (D × N). Due to this relationship, in equation 16, since the lane width can be considered to be the same when a vehicle is traveling, it is possible to suppress observation noise by applying smoothing to D1[n] × N1[n]. Furthermore, even in multiple consecutive frames, it is assumed that the vehicle is traveling in the same lane, so the actual lane width can be considered to be the same, and it is possible to suppress observation noise by applying smoothing to D1[n] × N1[n].

[0302] On the other hand, in some frames, the image may be blurred due to the vehicle running over small stones, potentially preventing the accurate acquisition of lane width data. Therefore, it may be possible to reduce the error from the reference data by reducing the weight of frames with a large discrepancy in D1[n]×N1[n] (i.e., large blur).

[0303] Alternatively, the calculation may be adjusted by increasing the weight as the time difference from the current frame decreases and decreasing the weight as the time difference increases, thereby balancing the response to the current frame with the smoothing of the fluctuations that occurred between frames. Furthermore, since the reference data remains almost constant during periods when the lane width does not change significantly, the system may be configured to reduce the processing load by using previously obtained reference data during such periods and skipping the above calculation (Equation 17).

[0304] In step S2904, the object distance calculation unit 2204 uses the reference data B obtained in the previous step and the lane width N2 of the object to be measured to determine the distance data D2 of the object to be measured. The distance data D2 is determined by the following equation (Equation 18).

[0305]

number

[0306] As described above, this embodiment describes an example of a distance measuring system that can measure distances with high accuracy from near to far by combining stereo distance measurement using parallax images and distance estimation from a single image. Even at long distances where distance measurement accuracy deteriorates with distance measurement using parallax images alone, distance accuracy can be maintained even when the object to be measured is far away by using the size ratio of scaling objects such as road surfaces.

[0307] When scaling targets objects whose actual size does not change significantly, such as roads, smoothing during scaling reduces observation noise during size measurement, thereby improving distance accuracy. Furthermore, by changing the blending ratio according to the distance range between distance measurement using disparity images and distance estimation using single images, more robust distance estimation becomes possible. In addition, by detecting and correcting the roll angle, robust distance estimation becomes possible, reducing the effects of vehicle deformation and road surface irregularities.

[0308] <Other Embodiments> In the embodiments described above, an imaging device was explained as an example in which left and right parallax images are acquired using a phase-difference imaging method via the same optical system to acquire distance image data. However, the method of acquiring parallax images is not limited to this. It is also possible to acquire left and right parallax images using a so-called stereo camera, in which a left parallax image and a right parallax image are acquired by two imaging devices placed at a predetermined distance apart.

[0309] Furthermore, it is also possible to perform the aforementioned distance measurement using distance measurement devices such as LiDAR to acquire distance information and external information obtained through image recognition of the captured images obtained from the imaging device. Furthermore, the above embodiments may be combined as appropriate. Furthermore, in the above embodiments, integrated distance information may be generated based on at least two histories of the first to third distance information of an object or the history of integrated distance information.

[0310] In the above-described embodiment, an example was given in which a distance calculation device as an electronic device was mounted on a vehicle, which was the mobile object. However, the mobile object can be any mobile object, such as a motorcycle, bicycle, wheelchair, ship, airplane, drone, or mobile robot such as an AGV or AMR. Furthermore, the distance calculation device as an electronic device in this embodiment is not limited to those mounted on such mobile objects, but also includes devices that acquire images from a camera or the like mounted on the mobile object via communication and calculate the distance at a location away from the mobile object.

[0311] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications are possible based on the spirit of the present invention, and these modifications are not excluded from the scope of the present invention.

[0312] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of Symbols]

[0313] 100: Vehicles 101: Driver 110: Imaging device 120: Radar equipment 130: Route generation ECU 140: Vehicle control ECU 150: Route Generator 160: Vehicle Information Measuring Instrument

Claims

1. A first distance information acquisition means that acquires first distance information by a phase difference distance measurement method based on the region corresponding to an object contained in the image signal, A second distance information acquisition means that acquires second distance information based on the positional relationship between the information of the lower end position of the object and the vanishing point information included in the image signal, A third distance information acquisition means that acquires third distance information based on the width or height information of the object included in the image signal, A fourth distance information acquisition means for acquiring a fourth distance information which is a distance measurement value estimated by a motion model estimated from velocity information including relative velocity, relative acceleration, and relative jerk parameters with respect to the object, An electronic device characterized by having distance information integrating means that generates integrated distance information by weighting and integrating the first distance information, second distance information, third distance information, and fourth distance information using weighting coefficients corresponding to each of the first distance information, second distance information, third distance information, and fourth distance information.

2. The electronic device according to Claim 1, characterized in that the first distance information acquisition means generates a frequency distribution of distance information corresponding to a plurality of pixels in the region, and acquires the most frequently occurring distance information in the frequency distribution as the first distance information.

3. The electronic device according to claim 1, characterized in that the first distance information acquisition means acquires the first distance information by a phase difference distance measurement method based on signals from a first photoelectric conversion unit and a second photoelectric conversion unit arranged within the pixels of an image sensor.

4. The electronic device according to claim 1, characterized in that the first distance information acquisition means acquires the first distance information by a phase difference distance measurement method based on two image signals from a stereo camera.

5. The electronic device according to claim 1, characterized in that the second distance information acquisition means estimates the roll angle of the camera that acquires the image signal.

6. The electronic device according to any one of claims 1 to 5, characterized in that the distance information integration means generates the integrated distance information based on the type of object.

7. The electronic device according to any one of claims 1 to 5, characterized in that the distance information integration means determines the weight coefficient based on the variance corresponding to each of the first to fourth distance information.

8. The electronic device according to claim 7, characterized in that the distance information integrating means determines the weight coefficient based on the variance such that the weight coefficient increases as the variance of the first to fourth distance information and the distance measurement value by the motion model decreases.

9. A first distance information acquisition step involves acquiring first distance information using a phase difference distance measurement method based on the region corresponding to an object contained in the image signal, A second distance information acquisition step, which acquires second distance information based on the positional relationship between the information of the lower end position of the object and the vanishing point information included in the image signal, A third distance information acquisition step, which acquires third distance information based on the width or height information of the object included in the image signal, A fourth distance information acquisition step, which involves acquiring a fourth distance information, which is a measured distance value estimated by a motion model that is estimated from velocity information including relative velocity, relative acceleration, and relative jerk parameters with respect to the object; A distance calculation method characterized by comprising: a distance information integration step of generating integrated distance information by weighting and integrating the first distance information, second distance information, third distance information, and fourth distance information using weight coefficients corresponding to each of the first distance information, second distance information, third distance information, and fourth distance information.

10. A computer program for controlling each means of an electronic device according to any one of claims 1 to 8 by a computer.

Images