Electronic apparatus, mobile device, distance calculation method, and storage medium

Abstract

The present application relates to an electronic apparatus, a mobile device, a distance calculation method, and a storage medium. The electronic apparatus includes: a first distance information acquisition unit configured to acquire first distance information corresponding to an object included in an image signal; at least one of a second distance information acquisition unit configured to acquire second distance information based on information of an end position of the object included in the image signal, and a third distance information acquisition unit configured to acquire third distance information based on information of a size of the object included in the image signal; and a distance information integration unit configured to generate integrated distance information by combining and integrating at least two of the first distance information, the second distance information, and the third distance information.

Description

Technical Field

[0001] This invention relates to electronic instruments, mobile devices, distance calculation methods, and storage media for obtaining distances from mobile devices to objects. Background Technology

[0002] An imaging device exists that has a sensor with multiple pixel regions having photoelectric conversion functions arranged in a two-dimensional manner, and is capable of acquiring image signals and distance information in each pixel region. In a solid-state imaging element disclosed in Japanese Patent Application Publication No. 2007-281296, pixels with ranging functions are arranged for some or all of the pixels in the imaging element, and the distance to the subject is detected based on the phase difference detected on the imaging surface (imaging surface phase difference system).

[0003] In other words, the positional deviation is calculated based on the correlation between two image signals generated from images based on light beams passing through different pupil regions of the camera optical system set in the imaging device, and the distance is obtained based on this positional deviation. A technique such as region-based matching is used to evaluate the correlation between the two image signals, which assesses the correlation by cutting out image signals included in a predetermined comparison region from each image signal.

[0004] However, due to the subject or shooting conditions (such as small variations in the contrast of the subject included in the image signal or the presence of a large amount of noise in the image signal), misjudgment of correlation may occur. If the occurrence of misjudgment of the included correlation reaches or exceeds a certain level, there is concern that the calculated positional deviation between the two image signals may be inaccurate, and the accuracy of the obtained distance may decrease. Summary of the Invention

[0005] According to one aspect of the present invention, an electronic instrument includes: at least one processor or circuit configured to function as: a first distance information acquisition unit for acquiring first distance information corresponding to an object included in an image signal; at least one of a second distance information acquisition unit and a third distance information acquisition unit, wherein the second distance information acquisition unit is configured to acquire second distance information based on at least one end position information of the object included in the image signal, and the third distance information acquisition unit is configured to acquire third distance information based on size information of the object included in the image signal; and a distance information integration unit for generating integrated distance information by combining and integrating at least two of the first distance information, the second distance information, and the third distance information.

[0006] Further features of the invention will become apparent from the following description of embodiments with reference to the accompanying drawings. Attached Figure Description

[0007] Figure 1 This is a schematic diagram illustrating an example of the construction of a vehicle according to the first embodiment.

[0008] Figure 2 This is a block diagram illustrating an example of the construction of a vehicle according to the first embodiment.

[0009] Figure 3 This is a functional block diagram illustrating an example of the construction of a path generation apparatus according to the first embodiment.

[0010] Figure 4A and Figure 4B This is a schematic diagram illustrating an example of the construction of a camera element according to the first embodiment.

[0011] Figures 5A to 5D This is an illustrative diagram showing the relationship between the distance to the subject and the incident light in a phase difference system of a camera plane.

[0012] Figure 6A and Figure 6B This is a flowchart illustrating an example of processing performed by the image processing unit according to the first embodiment.

[0013] Figure 7A and Figure 7B This is a flowchart illustrating an example of the processing performed by the distance information generation unit according to the first embodiment.

[0014] Figures 8A to 8E This is a schematic diagram illustrating an example of an image and information in a processing example performed by the distance information generation unit according to the first embodiment.

[0015] Figure 9 This is a flowchart illustrating an example of the path generation process performed by the path generation unit according to the first embodiment.

[0016] Figure 10 This is a flowchart illustrating an example of object distance information generation processing performed by the distance information generation unit according to Modified Example 2.

[0017] Figure 11 This is an illustrative diagram illustrating the time-varying object distance information of an object with the same identification number as the Nth object.

[0018] Figure 12 This is a flowchart illustrating an example of object distance information generation processing performed by the distance information generation unit according to Modified Example 3.

[0019] Figures 13A to 13D This is an illustrative diagram of a weighted average.

[0020] Figure 14 This is a flowchart illustrating an example of a process for obtaining distance values ​​(object distance information) according to a second embodiment.

[0021] Figure 15 This is a flowchart illustrating a portion of a processing example of multiple consecutive input data in a time series according to a second embodiment.

[0022] Figure 16 It is shown Figure 15 The flowchart shows the rest of the processing example.

[0023] Figure 17A and Figure 17B This is an illustrative schematic diagram illustrating an example of absolute error correction processing according to the second embodiment.

[0024] Figure 18 This is a block diagram illustrating an example of the construction of a ranging system according to a third embodiment.

[0025] Figure 19 This is a schematic diagram illustrating an example of the output results of the segmentation line detection task and the object recognition task performed by the object detection unit according to the third embodiment.

[0026] Figure 20 This is an explanatory diagram illustrating an example of the positional relationship between the camera attachment position and the road surface according to the third embodiment.

[0027] Figure 21 This is an illustration of an example scenario where two signs with known object sizes are detected on a road.

[0028] Figure 22 This is a block diagram illustrating the construction of a modified example of a scaling distance measuring unit.

[0029] Figures 23A to 23D This is a schematic diagram illustrating an example of the processing of lane width detection and lane center detection performed by the lane analysis unit according to the third embodiment.

[0030] Figure 24 This is a schematic diagram showing the comparison of the presence or absence of a lateral tilt angle between the positions of the width of the traffic lanes in the captured image.

[0031] Figure 25 This is a flowchart illustrating an example of the roll angle estimation process performed by the roll angle estimation unit according to the third embodiment.

[0032] Figures 26A to 26C These are illustrative schematic diagrams illustrating various processing examples in the roll angle estimation process according to the third embodiment.

[0033] Figure 27This is a flowchart illustrating an example of coordinate estimation processing of the lane width data performed by the grounding position estimation unit according to the third embodiment.

[0034] Figure 28 This is a schematic diagram illustrating an example of the coordinates of the grounding position of the ranging target set by the grounding position estimation unit according to the third embodiment.

[0035] Figure 29 This is a flowchart illustrating an example of the distance estimation process to the ranging target performed by the object distance calculation unit according to the third embodiment.

[0036] Figure 30 This is a schematic diagram illustrating an example of the width data of each traffic lane according to the third embodiment. Detailed Implementation

[0037] In the following description, with reference to the accompanying drawings, advantageous embodiments will be used to illustrate the invention. In the drawings, the same reference numerals are applied to the same components or elements, and repeated descriptions will be omitted or simplified.

[0038] <First Embodiment>

[0039] In the following description, a path generation device (electronic instrument) including a shooting device will be used as an example of the path generation device of the present invention, but the application of the present invention is not limited thereto.

[0040] In the description of the accompanying drawings, even if the drawing numbers are different, the same reference numerals will be applied to the parts that indicate the same locations, and repetitive descriptions will be avoided as much as possible.

[0041] Figure 1 This is a schematic diagram illustrating an example of the construction of a vehicle 100 according to the first embodiment. The vehicle 100 is an example of a mobile device including a camera device 110, a radar device 120, a path generation ECU 130, a vehicle control ECU 140, and a measuring instrument group 160. The vehicle 100 also includes a drive unit 170, a memory 180, and a memory 190, which serve as a drive control unit for driving the vehicle 100.

[0042] Will use Figure 2 This will be used to explain drive unit 170, memory 180, and memory 190. In Figure 2 It includes a camera device 110, a radar device 120, a path generation ECU 130, a vehicle control ECU 140, and a measuring instrument group 160.

[0043] The camera device 110 and the path generation ECU 130 constitute the path generation device 150. A driver 101 may ride in the vehicle 100, and the driver 101 rides in the vehicle 100 facing forward (in the direction of travel) while it is in motion. The driver 101 can control the operation of the vehicle 100 by operating control components such as the steering wheel, accelerator pedal, and brake pedal. The vehicle 100 may have automatic driving capabilities or may be remotely controllable from an external source.

[0044] The camera device 110 is positioned to film the front side (normal driving direction) of the vehicle 100. For example... Figure 1 As shown, the camera device 110 is arranged, for example, near the upper end of the windshield in the vehicle 100, and shoots an area within a predetermined angle range (hereinafter referred to as the shooting angle) towards the front side of the vehicle 100.

[0045] The camera device 110 can be arranged to film the rear side of the vehicle 100 (opposite to the normal driving direction, i.e., the reverse driving direction), or it can be arranged to film the side side. Multiple camera devices 110 can be arranged in the vehicle 100.

[0046] Figure 2 This is a block diagram illustrating an example of the construction of a vehicle 100 according to the first embodiment.

[0047] The camera device 110 captures images of the environment surrounding the vehicle, including the road (driving road) on which the vehicle 100 travels, and detects objects within the field of view of the camera device 110. Additionally, the camera device 110 acquires information about the detected objects (external information) and information related to the distance to the detected objects (object distance information), and outputs both pieces of information to the path generation ECU 130.

[0048] The object distance information only needs to be information that can be converted into distance from a predetermined position in the vehicle 100 to the object using a predetermined reference table or predetermined conversion coefficients and conversion expressions. For example, the distance can be assigned to predetermined integer values ​​and output sequentially to the path generation ECU 130.

[0049] The camera device 110 has a sensor with multiple pixel regions arranged in a two-dimensional manner and having photoelectric conversion function, and can obtain the distance of the object through a camera plane phase difference system. The acquisition of the object's distance information through the camera plane phase difference system will be explained below.

[0050] Radar device 120 is a detection device for detecting objects by emitting electromagnetic waves and receiving their reflected waves. Radar device 120 serves as a fourth range information acquisition unit, which acquires range information (fourth range information) representing the distance to an object in the direction of electromagnetic wave transmission based on the time from the emission of the electromagnetic wave until the reception of the reflected wave and the received intensity of the reflected wave.

[0051] Radar device 120 outputs range information to path generation ECU 130. It is assumed that radar device 120 is a millimeter-wave radar device that uses electromagnetic waves with wavelengths in the so-called band range from millimeter-wave to submillimeter-wave.

[0052] In this embodiment, multiple radar devices 120 are attached to the vehicle 100. For example, the radar devices 120 are attached to each of the left front side, right front side, left rear side, and right rear side of the vehicle 100.

[0053] In addition, each radar device 120 emits electromagnetic waves within a predetermined angle range. Based on the time from the emission of the electromagnetic waves until the received reflected waves are obtained, and the received intensity of the reflected waves, the distance relative to each radar device 120 is measured, and distance information about the object is generated. The distance information may include, in addition to information related to the distance relative to the radar device 120, information related to the received intensity of the reflected waves and the relative velocity of the object.

[0054] For example, the measuring instrument group 160 includes a speed measuring instrument 161, a steering angle measuring instrument 162, and an angular velocity measuring instrument 163. Through these three measuring instruments, vehicle information related to the driving status of the vehicle (such as speed, steering angle, and angular velocity) is obtained. The speed measuring instrument 161 is used to detect the speed of the vehicle 100.

[0055] Steering angle measuring instrument 162 is used to detect the steering angle of vehicle 100. Angular velocity measuring instrument 163 is used to detect the angular velocity of vehicle 100 in the turning direction. Each measuring instrument outputs the measured signal corresponding to the parameter as vehicle information to path generation ECU 130.

[0056] The path generation ECU 130 is constructed using logic circuits and other components, and generates the driving trajectory of the vehicle 100 and path information related to the driving trajectory based on measurement signals, external information, object distance information, and distance information. The path generation ECU 130 outputs the driving trajectory and path information to the vehicle control ECU 140. The data and programs processed and executed by the path generation ECU 130 are stored in the memory 180.

[0057] Here, the driving trajectory refers to the information indicating the path (track) traversed by the vehicle 100. Additionally, the path information is the information (including road information, etc.) used by the vehicle 100 to traverse the path indicated by the driving trajectory.

[0058] The vehicle control ECU 140 is constructed using logic circuits and controls the drive unit 170, enabling the vehicle 100 to traverse a path corresponding to the path information based on path information and vehicle information obtained from the measuring instrument group 160. The data and programs processed and executed by the vehicle control ECU 140 are stored in the memory 190.

[0059] The drive unit 170 is a drive component for driving a vehicle, and includes, for example, a power unit (not shown) such as an engine or motor that generates energy to rotate the tires, and a steering unit for controlling the direction of travel of the vehicle. Additionally, the drive unit 170 includes a gearbox for rotating the tires using the energy generated by the power unit, a gear control unit for controlling the internal structure of the gearbox, and a braking unit for performing braking operations.

[0060] The vehicle control ECU 140 adjusts the driving amount, braking amount, and steering amount of the vehicle 100 by controlling the drive unit 170, so that the vehicle travels along a path corresponding to the path information. Specifically, the vehicle control ECU 140 controls the braking, steering, and gear mechanism to operate the vehicle 100.

[0061] The route generation ECU 130 and the vehicle control ECU 140 can be configured with a common central processing unit (CPU) and memory storing the calculation program. The HMI 240 is a human-machine interface used to provide information to the driver 101.

[0062] HMI 240 includes a display that can be visually recognized when the driver 101 is in the driving position, and a display control device for generating information to be displayed on the display. Additionally, HMI 240 includes a device for outputting audio (speaker system) and an audio control device for generating audio data.

[0063] The display control unit within the HMI 240 enables the display to show navigation information based on the route information generated by the route generation ECU 130. Additionally, the audio control unit within the HMI 240 generates audio data based on the route information to inform the driver 101 of the route information and outputs this audio data from the speaker system. For example, the audio data might be used to notify the driver of an approaching intersection.

[0064] Figure 3This is a functional block diagram illustrating an example of the construction of a path generation apparatus 150 according to a first embodiment. The path generation apparatus 150 includes a camera device 110 and a path generation ECU 130. This is achieved by causing a computer included in the path generation apparatus 150 to execute a computer program stored in a memory that serves as a storage medium. Figure 3 Part of the function block shown.

[0065] However, some or all of these functional blocks can be implemented in hardware. This can be done using dedicated application-specific integrated circuits (ASICs) or processors (reconfigurable processors, DSPs, etc.).

[0066] in addition, Figure 3 The functional block diagram shown may not be housed within the same housing and can consist of separate devices interconnected via signal lines. Figure 3 The above-mentioned explanation also applies in a similar manner to [the following]: Figure 18 and Figure 22 Explanation.

[0067] The camera device 110 includes a camera optical system 301, a camera element 302, an image processing unit 310, and an object information generation unit 320. In this embodiment, the camera optical system 301, the camera element 302, the image processing unit 310, and the object information generation unit 320 are arranged inside the housing (not shown) of the camera device 110.

[0068] The camera optical system 301 includes a capturing lens of the camera device 110 and has the function of forming an image (optical image) of a subject on the imaging element 302. The camera optical system 301 is composed of multiple lens groups. The camera optical system 301 has an exit pupil at a predetermined distance from the imaging element 302.

[0069] The imaging element 302 is composed of a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) and has a ranging function performed by a phase difference ranging method of the imaging surface. In the imaging element 302, multiple pixel regions with photoelectric conversion function are arranged in two dimensions.

[0070] For example, each pixel region has two photoelectric conversion units (a first photoelectric conversion unit and a second photoelectric conversion unit) arranged separately along the row direction. For example, a color filter of R, G, and B, and a microlens are arranged in front of the two photoelectric conversion units in each pixel region.

[0071] The imaging element 302 performs photoelectric conversion on the subject image formed on the imaging element 302 via the imaging optical system 301, generates an image signal based on the subject image, and outputs the image signal to the image processing unit 310. The image signal is a signal with an output value for each photoelectric conversion unit in each pixel area.

[0072] The imaging element 302 outputs a first image signal based on a signal output from the first photoelectric conversion unit and a second image signal based on a signal output from the second photoelectric conversion unit. Alternatively, the summed signal obtained by adding the first image signal and the second image signal, as well as the first image signal, are output to the image processing unit 310.

[0073] The image processing unit 310 generates image data having brightness information for each color (red, green, and blue) for each pixel and distance image data representing distance information for each pixel based on the image signal supplied from the imaging element 302. The image processing unit 310 includes: an imaging unit 311 that generates image data based on the image signal supplied from the imaging element 302; and a distance image generation unit 312 that generates distance image data based on the image signal supplied from the imaging element 302.

[0074] The following will describe the processing performed by these units. The image processing unit 310 outputs image data from the display unit 311 and distance image data from the distance image generation unit 312 to the object information generation unit 320.

[0075] The object information generation unit 320 includes an identification processing unit 321, which detects objects included in the acquired image and generates external information representing information related to the objects based on image data from the display unit 311. The external information includes information such as the position of the detected object within the image, its size (e.g., width or height), and its region. Additionally, the external information includes information related to the attributes and identification number of the detected object.

[0076] The object information generation unit 320 includes a distance information generation unit 322, which generates object distance information (first distance information) representing the distance to objects included in the acquired image based on external information from the identification processing unit 321 and distance image data from the distance image generation unit 312. Here, the distance information generation unit 322 serves as a first distance information acquisition unit for acquiring first distance information corresponding to objects included in the image signal.

[0077] Furthermore, the object distance information is linked to information related to the identification number of the object included in the external information. The object information generation unit 320 outputs the external information from the identification processing unit 321 and the object distance information from the distance information generation unit 322 to the path generation ECU 130.

[0078] The image processing unit 310 and the object information generation unit 320 can be composed of one or more processors provided in the camera device 110. The functions of the image processing unit 310 and the object information generation unit 320 can be implemented by having one or more processors execute programs read from the memory 340.

[0079] The path generation ECU 130 includes a path generation unit 330. The path generation unit 330 generates path information based on external information, object distance information, and distance information acquired from the radar device 120.

[0080] Next, the structure and control of each block in the path generation device 150 will be described in detail.

[0081] Figure 4A and Figure 4B This is a schematic diagram illustrating an example of the construction of the imaging element 302 according to the first embodiment. Figure 4A This is a top view of the camera element 302 as seen from the direction of light incidence. The camera element 302 is constructed by arranging multiple pixel groups 410 (2 rows × 2 columns) in a matrix shape.

[0082] Each pixel group 410 has green pixels G1 and G2 for detecting green light, red pixel R for detecting red light, and blue pixel B for detecting blue light. In pixel group 410, green pixels G1 and G2 are arranged diagonally. Additionally, each pixel has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412.

[0083] Figure 4B It is along Figure 4A A cross-sectional view of pixel group 410 along line I-I'. Each pixel is composed of a microlens 413, a light guide layer 414, and a light receiving layer 415. The light guide layer 414 includes a microlens 413 for efficiently guiding light beams incident on the pixel to the light receiving layer 415, a color filter for allowing light with a wavelength band corresponding to the color of the light detected by each pixel to pass through, and wiring for image reading and wiring for pixel driving.

[0084] The light-receiving layer 415 is a photoelectric conversion unit that performs photoelectric conversion on light incident through the light-guiding layer 414 and outputs the light as an electrical signal. The light-receiving layer 415 has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412.

[0085] Figures 5A to 5D This is an illustrative diagram showing the relationship between the distance to the subject and the incident light in a phase difference system of a camera plane. Figure 5A This is a schematic diagram illustrating the exit pupil 501 of the imaging optical system 301, the green pixel G1 of the imaging element 302, and the light incident on each photoelectric conversion unit of the green pixel G1. The imaging element 302 has multiple pixels. However, for simplicity, only one green pixel G1 will be described.

[0086] The microlens 413 of the green pixel G1 is arranged such that the exit pupil 501 and the light-receiving layer 415 are optically conjugate. As a result, a light beam passing through the first pupil region 510, which is a portion of the pupil region involved in the exit pupil 501, is incident on the first photoelectric conversion unit 411. Similarly, a light beam passing through the second pupil region 520, which is a portion of the pupil region, is incident on the second photoelectric conversion unit 412.

[0087] Each pixel's first photoelectric conversion unit 411 performs photoelectric conversion on the received light beam and outputs a signal. A first image signal is generated based on the signals output from the plurality of first photoelectric conversion units 411 included in the imaging element 302. The first image signal represents the intensity distribution of the image (hereinafter referred to as image A) formed on the imaging element 302 by the light beam that mainly passes through the first pupil region 510.

[0088] Each pixel's second photoelectric conversion unit 412 performs photoelectric conversion on the received light beam and outputs a signal. A second image signal is generated based on the signals output from the plurality of second photoelectric conversion units 412 included in the imaging element 302. The second image signal represents the intensity distribution of the image (hereinafter referred to as image B) formed on the imaging element 302 by the light beam that mainly passes through the second pupil region 520.

[0089] The amount of relative positional deviation between the first image signal corresponding to image A and the second image signal corresponding to image B (hereinafter referred to as the disparity) is a quantity corresponding to the defocus amount. This will be used... Figure 5B , Figure 5C and Figure 5D This will illustrate the relationship between parallax and defocus.

[0090] Figure 5B , Figure 5C and Figure 5D This is a schematic diagram illustrating the imaging element 302 and the imaging optical system 301. In the figure, reference numeral 511 indicates a first light beam passing through a first pupil region 510, and reference numeral 521 indicates a light beam passing through a second pupil region 520.

[0091] Figure 5BThe image is shown in the focusing state, and the first beam 511 and the second beam 521 converge on the imaging element 302. At this time, the parallax (positional deviation) between the first image signal corresponding to the image A formed by the first beam 511 and the second image signal corresponding to the image B formed by the second beam 521 becomes zero.

[0092] Figure 5C This example illustrates a state of defocus on the image side in the negative z-axis direction. In this case, the disparity between the first image signal formed by the first beam 511 and the second image signal formed by the second beam 521 does not become zero and has a negative value.

[0093] Figure 5D This example illustrates a state of defocus on the positive z-axis on the image side. In this case, the disparity between the first image signal formed by the first beam 511 and the second image signal formed by the second beam 521 does not become zero and has a positive value.

[0094] according to Figure 5C and Figure 5D The comparison confirmed that the direction of parallax shifts between positive and negative values ​​depending on the amount of defocus. Furthermore, based on geometric relationships, the parallax amount is confirmed to be determined by the amount of defocus.

[0095] Therefore, as described below, the disparity between the first and second image signals can be detected using a region-based matching technique, and the disparity can be converted into defocus via a predetermined conversion coefficient. Furthermore, the image-side defocus can be converted into a distance to the object using the imaging formula of the camera optical system 301.

[0096] As described above, the imaging element 302 can output the sum of the first image signal and the second image signal (composite signal) and the first image signal to the image processing unit 310 respectively. In this case, the image processing unit 310 can generate the second image signal by the difference between the sum of the signal (composite signal) and the first image signal, so that the first image signal and the second image signal can be acquired separately.

[0097] Next, the processing performed by the image processing unit 310 will be explained. Figure 6A and Figure 6B This is a flowchart illustrating an example of processing performed by the image processing unit 310 according to the first embodiment. The processing is performed by causing the computer included in the path generation apparatus 150 to execute a computer program stored in its memory. Figure 6A and Figure 6B The flowchart shows the steps involved.

[0098] Figure 6AThis is a flowchart illustrating the display processing operation of the display unit 311 of the image processing unit 310, which generates image data based on image signals (a first image signal and a second image signal). The display processing is performed when an image signal is received from the imaging element 302.

[0099] In step S601, the imaging unit 311 performs processing to generate a composite image signal, which is achieved by combining a first image signal and a second image signal input from the imaging element 302. An image signal based on an image formed by a light beam passing through the entire area of ​​the exit pupil 501 can be obtained by combining the first image signal and the second image signal.

[0100] When the pixel coordinate of the camera element 302 in the horizontal direction is x and its pixel coordinate in the vertical direction is y, the composite image signal Im(x,y) of pixel (x,y) can be represented by the first image signal Im1(x,y) and the second image signal Im2(x,y) through the following expression 1.

[0101] Im(x, y) = Im1(x, y) + Im2(x, y)...(Expression 1)

[0102] In step S602, the display unit 311 performs correction processing on defective pixels in the synthesized image signal. Defective pixels are pixels in the imaging element 302 that cannot output normal signals. The coordinates of the defective pixels are pre-stored in the memory. The display unit 311 retrieves information representing the coordinates of the defective pixels in the imaging element 302 from the memory.

[0103] The imaging unit 311 performs correction by replacing the median filter of the composite image signal of the defective pixel with the median of the composite image signal of the pixels surrounding the defective pixel. Regarding the method for correcting the composite image signal of the defective pixel, the signal value of the defective pixel can be generated by interpolation using the signal values ​​of the pixels surrounding the defective pixel, with the aid of pre-prepared coordinate information of the defective pixel.

[0104] In step S603, the imaging unit 311 applies a light amount correction process to the synthesized image signal to correct the reduction in light amount around the field angle caused by the imaging optical system 301. The characteristic (relative light amount ratio) of the reduction in light amount around the field angle caused by the imaging optical system 301 is pre-stored in the memory.

[0105] Regarding the method for correcting the amount of light, the amount of light can be corrected by reading the relative light amount ratio between pre-stored field angles from the memory and multiplying the synthesized image signal by a gain that makes the light amount ratio constant. For example, the imaging unit 311 performs light amount correction by multiplying the synthesized image signal of each pixel by a gain that has the characteristic of increasing from the central pixel in the imaging element 302 toward the surrounding pixels.

[0106] In step S604, the imaging unit 311 performs noise reduction processing on the synthesized image signal. Noise reduction using a Gaussian filter can be used as a method for noise reduction.

[0107] In step S605, the imaging unit 311 performs demosaic processing on the synthesized image signal, acquiring red (R), green (G), and blue (B) color signals for each pixel, and using these color signals to generate image data containing the brightness information of each pixel. Regarding the demosaic method, a technique can be used to interpolate the color information of each pixel using linear interpolation for each color channel.

[0108] In step S606, the display unit 311 performs grayscale correction (gamma correction processing) using a predetermined gamma value. The image data Id(x,y) of the pixel (x,y) before grayscale correction and the gamma value γ are used to represent the image data Idc(x,y) of the pixel (x,y) after grayscale correction using the following expression 2.

[0109] Idc(x, y) = Id(x, y) γ ...(Expression 2)

[0110] A pre-prepared value can be used as the gamma value γ. The gamma value γ can be determined based on the position of the pixel. For example, the gamma value γ can be changed for each region obtained by dividing the effective area of ​​the imaging element 302 into a predetermined number of segments.

[0111] In step S607, the display unit 311 performs a color space conversion process to convert the color space of the image data from RGB color space to YUV color space. The display unit 311 uses predetermined coefficients and a color space conversion expression (mathematical expression 3) to convert the image data corresponding to the brightness of each color (red, green, and blue) into brightness values ​​and color difference values, and converts the color space of the image data from RGB color space to YUV color space.

[0112] In mathematical expression 3, factor IdcR(x,y) represents the red image data value of pixel (x,y) after grayscale correction. Factor IdcG(x,y) represents the green image data value of pixel (x,y) after grayscale correction. Factor IdcB(x,y) represents the blue image data value of pixel (x,y) after grayscale correction. Factor Y(x,y) represents the luminance value of pixel (x,y) obtained through color space conversion. Factor U(x,y) represents the difference (color difference value) between the luminance value and the blue component of pixel (x,y) obtained through color space conversion.

[0113] The factor V(x, y) represents the difference (color difference value) between the luminance value and the red component of the pixel (x, y) obtained through color space conversion. The coefficients (ry, gy, and by) are used to obtain Y(x, y), and the coefficients (ru, gu, and bu) and (rv, gv, and bv) are each used to calculate the color difference value.

[0114] Y(x,y)=ry×IdcR(x,y)+gy×IdcG(x,y)+by×IdcB(x,y)

[0115] U(x,y)=ru×IdcR(x,y)+gu×IdcG(x,y)+bu×IdcB(x,y)

[0116] V(x, y)=rv×IdcR(x, y)+gv×IdcG(x, y)+bv×IdcB(x, y)....(Expression 3)

[0117] In step S608, the imaging unit 311 performs correction (distortion correction) on the converted image data to suppress the effects of distortion aberrations caused by the optical characteristics of the imaging optical system 301. Distortion aberration correction is performed by geometrically deforming the image data to correct the distortion rate of the imaging optical system 301.

[0118] Geometric deformation is performed using a polynomial expression to generate the pre-correction pixel positions based on the correct pixel positions without distortion. If the pre-correction pixel positions are represented as decimals, the nearest pixel can be used after rounding, or linear interpolation can be used.

[0119] In step S609, the imaging unit 311 outputs the image data with distortion and aberration correction applied to the object information generation unit 320. Thus, the imaging process performed by the imaging unit 311 ends. However, this process is repeated periodically. Figure 6A The process continues until the user issues an instruction to end (not shown).

[0120] If the recognition processing unit 321 can generate external information through external recognition processing based on the image data obtained from the imaging element 302 before various correction processing, the display unit 311 may be unable to perform the following: Figure 6A All processing in the process. For example, if the identification processing unit 321 can detect objects within the shooting field angle range based on image data without applying the distortion correction processing of step S608, then the processing of step S608 can be removed from... Figure 6A The image processing is omitted.

[0121] Figure 6B This is a flowchart illustrating the operation of the distance image generation unit 312 in generating distance image data. Here, distance image data is data associated with each pixel, corresponding to distance information from the camera device 110 to the subject. The distance information may be a distance value D, or it may be a defocus amount ΔL or a parallax amount d used to calculate the distance value. In this embodiment, the distance image data will be described as data that maps the distance value D to each pixel.

[0122] In step S611, the distance image generation unit 312 generates a first brightness image signal and a second brightness image signal based on the input image signal. That is, the distance image generation unit 312 uses the first brightness image signal corresponding to image A to generate the first image signal, and uses the second image signal corresponding to image B to generate the second brightness image signal.

[0123] Assume that the distance image generation unit 312 uses coefficients to synthesize the image signal values ​​of the red, green, and blue pixels in each pixel group 410, and generates a brightness image signal. The distance image generation unit 312 can generate a brightness image signal by synthesizing the red, green, and blue channels by multiplying them by predetermined coefficients after performing demosaic processing using linear interpolation.

[0124] In step S612, the distance image generation unit 312 corrects the light balance between the first luminance image signal and the second luminance image signal. The light balance correction is performed by multiplying at least one of the first luminance image signal and the second luminance image signal by a correction coefficient.

[0125] Assuming that a correction coefficient is pre-calculated, uniform illumination is applied after the positions of the camera optical system 301 and the camera element 302 are adjusted, and the brightness ratio between the obtained first brightness image signal and the second brightness image signal becomes constant, the correction coefficient is stored in the memory 340.

[0126] The distance image generation unit 312 generates a first image signal and a second image signal that have undergone light balance correction by multiplying at least one of the first brightness image signal and the second brightness image signal by a correction coefficient read from the memory.

[0127] In step S613, the distance image generation unit 312 performs noise reduction processing on the first and second luminance image signals that have undergone light balance correction. The distance image generation unit 312 performs noise reduction processing by applying a low-pass filter to each luminance image signal to reduce the high spatial frequency band.

[0128] The distance image generation unit 312 can also use a bandpass filter that allows transmission of a predetermined spatial frequency band. In this case, the effect of reducing the influence of correction error in the light balance correction performed in step S612 can be achieved.

[0129] In step S614, the distance image generation unit 312 calculates a disparity quantity, which is a measure of the relative positional deviation between the first brightness image signal and the second brightness image signal. The distance image generation unit 312 sets a focal point within the first brightness image corresponding to the first brightness image signal and sets a reference region centered on this focal point. Next, the distance image generation unit 312 sets a reference point within the second brightness image corresponding to the second brightness image signal and sets a reference region centered on this reference point.

[0130] The distance image generation unit 312 calculates the correlation between a first brightness image included in the control area and a second brightness image included in the reference area while progressively moving the reference point, and uses the reference point with the highest correlation as the corresponding point. The distance image generation unit 312 uses the amount of relative positional deviation between the focal point and the corresponding point as the disparity at the focal point.

[0131] The distance image generation unit 312 can calculate the disparity at multiple pixel locations by calculating the disparity while progressively moving the focus. As described above, the distance image generation unit 312 identifies a value representing the disparity value for each pixel and generates disparity image data as data representing the disparity distribution.

[0132] As a method for calculating the correlation used to obtain the disparity quantity using the distance image generation unit 312, known techniques can be used. For example, the distance image generation unit 312 can use the so-called normalized cross-correlation technique used to evaluate the normalized cross-correlation (NCC) between brightness images.

[0133] Additionally, the distance image generation unit 312 may also use techniques for evaluating dissimilarity as correlation. For example, the distance image generation unit 312 may use the sum of absolute differences (SAD) to evaluate the sum of absolute values ​​of the differences between brightness images or the sum of squared differences (SSD) to evaluate the sum of squares of the differences.

[0134] In step S615, the distance image generation unit 312 obtains the defocus amount of each pixel by converting the disparity amount of each pixel in the disparity image data into the defocus amount. The distance image generation unit 312 generates defocus image data representing the defocus amount of each pixel based on the disparity amount of each pixel in the disparity image data.

[0135] The distance image generation unit 312 uses the disparity d(x,y) of pixel (x,y) in the disparity image data and the conversion coefficient K to calculate the defocus amount ΔL(x,y) of pixel (x,y) according to the following expression 4. In the camera optical system 301, a portion of the first beam 511 and the second beam 521 is cut out in the surrounding field angle by vignetting. For this reason, the conversion coefficient K becomes a value that depends on the field angle (the position of the pixel).

[0136] ΔL(x, y)=K(x, y)×d(x, y)....(Expression 4)

[0137] If the imaging optical system 301 has the characteristic of image plane curvature that includes the focal position varying between the central field angle and the peripheral field angle, then when the image plane curvature is Cf, the parallax amount d(x, y) can be converted into defocus amount ΔL(x, y) using the following expression 5. The conversion coefficient K and the image plane curvature Cf can be obtained by acquiring the relationship between the parallax amount and the distance value relative to the object through graphical imaging after the imaging optical system 301 and the imaging element 302 are aligned. At this time, the image plane curvature Cf depends on the field angle and is given as a function of the pixel position.

[0138] ΔL(x, y)=K(x, y)×d(x, y)×Cf(x, y)....(Expression 5)

[0139] In step S616, the distance image generation unit 312 generates distance image data by converting the defocus amount ΔL(x,y) of pixel (x,y) into a distance value D(x,y) relative to an object in pixel (x,y). The distance value D relative to the object can be calculated by converting the defocus amount ΔL using the imaging relationship of the camera optical system 301.

[0140] When the focal length of the imaging optical system 301 is f and the distance from the image principal point to the imaging element 302 is Ipp, the defocus amount ΔL(x,y) can be converted into a distance value D(x,y) relative to the object using the imaging formula of the following expression 6.

[0141]

[0142] The focal length f and the distance Ipp from the image-side principal point to the imaging element 302 have constant values ​​independent of the field angle, but neither is limited to this. If the imaging magnification of the imaging optical system 301 varies significantly with respect to each field angle, then at least one of the focal length f and the distance Ipp from the image-side principal point to the imaging element 302 can have values ​​that vary significantly with respect to each field angle.

[0143] In step S617, the distance image generation unit 312 outputs the distance image data generated as described above to the object information generation unit 320. Thus, the distance image data generation process performed by the distance image generation unit 312 is completed.

[0144] However, repeating periodically Figure 6B The process continues until the user issues an end instruction (not shown). The parallax amount d, defocus amount ΔL, and distance D of the camera optical system 301 relative to the principal point for each pixel are values ​​that can be converted using the coefficients and conversion expressions described above.

[0145] Therefore, information representing the disparity d or defocus ΔL of each pixel can be provided as distance image data generated by the distance image generation unit 312. Considering that the object information generation unit 320 calculates representative values ​​of the distance values ​​D included in the object region, it is desirable to generate distance image data based on the defocus amount, which makes the frequency distribution symmetrical.

[0146] As described above, in the process of calculating the disparity in step S614, the correlation between the first luminance image and the second luminance image is used to search for corresponding points. However, if the first image signal contains a large amount of noise (e.g., noise caused by optical shot noise), or if the signal value of the luminance image signal included in the comparison area varies little, the correlation may not be accurately assessed.

[0147] In this case, a disparity value with a large error relative to the correct disparity value can be calculated. If the disparity value has a large error, the error of the distance value D generated in step S616 also increases.

[0148] Therefore, the distance image data generation process performed by the distance image generation unit 312 may include a reliability calculation process for calculating the reliability of the disparity quantity (disparity reliability). Disparity reliability is an indicator that represents how much error is included in the calculated disparity quantity. For example, the ratio of the standard deviation to the average signal value included in the control area can be used as the disparity reliability for evaluation.

[0149] If the change in signal value within the control area (so-called contrast) is significant, the standard deviation increases. If a large amount of light is incident on a pixel, the average value increases. If a large amount of light is incident on a pixel, there is a large amount of optical shot noise. In other words, the average value is positively correlated with the amount of noise.

[0150] The ratio of the mean to the standard deviation (standard deviation / mean) corresponds to the ratio between the contrast and noise levels. If the contrast is sufficiently large relative to the noise level, the error in the calculation of the disparity can be estimated to be small. In other words, it can be said that the greater the reliability of the disparity, the smaller the error in the calculated disparity and the more accurate the disparity.

[0151] Therefore, in step S614, reliability data representing the accuracy of the distance values ​​for each pixel constituting the distance image data can be generated by calculating the disparity reliability at each focal point. The distance image generation unit 312 can output the reliability data to the object information generation unit 320.

[0152] Next, the process by which the object information generation unit 320 generates external information and object distance information based on image data and distance image data will be explained.

[0153] Figure 7A and Figure 7B This is a flowchart illustrating an example of the processing performed by the object information generation unit 320 according to the first embodiment, and Figures 8A to 8E This is a schematic diagram illustrating an example of an image and information in a processing example performed by the object information generation unit 320 according to the first embodiment.

[0154] By causing the computer program stored in the computer execution memory included in the path generation device 150 to perform... Figure 7A and Figure 7B The flowchart shows the steps involved.

[0155] The identification processing unit 321 generates region information representing the location, size (such as width or height), and area of ​​objects included in the image based on image data, as well as external information representing the object's type (attribute) and identification number (ID number). The identification number is identification information used to identify the detected object, and is not limited to a number.

[0156] The identification processing unit 321 detects the types of objects present in the shooting field of the camera device 110, as well as the position and size of the objects in the image, determines whether the object has been registered, and assigns an identification number to the object.

[0157] Figure 8A An image 810 is shown, acquired by the camera device 110 and based on image data input to the object information generation unit 320. Image 810 includes a person 801, a vehicle 802, a sign 803, a road 804, and a traffic lane 805.

[0158] The identification processing unit 321 detects objects from the image 810 and generates external information representing the type, identification number, and region information of each object. Figure 8B It is a schematic diagram representing the external information of each object detected from image 810 at various locations in the xy coordinate plane.

[0159] For example, external information is generated as a table as shown in Table 1. In the external information, it is assumed that the region of an object is defined as a rectangular box (object box) that surrounds the object. In the external information, the region information of the object represents the shape of the rectangular object box as the coordinates (x0, y0) of the upper left side and the coordinates (x1, y1) of the lower right side.

[0160] [Table 1]

[0161]

[0162] Figure 7A This is a flowchart illustrating an example of the process by which the identification processing unit 321 generates external information. For example, the identification processing unit 321 begins the process of generating external information based on acquired image data.

[0163] In step S701, the identification processing unit 321 generates image data for object detection processing based on the image data. That is, the identification processing unit 321 performs processing to increase or decrease the size of the image data input from the image processing unit 310 to a size determined based on the detection performance and processing time in the object detection processing.

[0164] In step S702, the identification processing unit 321 performs processing for detecting objects included in the image based on image data, and detects the regions corresponding to the objects in the image and the types of objects. The identification processing unit 321 can detect multiple objects from one image. In this case, the identification processing unit 321 identifies the types and regions of the detected multiple objects.

[0165] The identification processing unit 321 generates the location and size (horizontal width and vertical height) of the region of the image where the object is detected, as well as the type of the object, as external information. For example, it is assumed that the types of objects that can be detected by the identification processing unit 321 include vehicles (passenger cars, buses, or trucks), people, animals, two-wheeled vehicles, and signs, etc.

[0166] The identification processing unit 321 detects objects and identifies the types of detected objects by comparing a predetermined external shape pattern associated with the type of object with the external shape of the object in the image. The types of objects that can be detected by the identification processing unit 321 are not limited to those mentioned above. However, from the viewpoint of processing speed, it is desirable to reduce the number of types of objects detected based on the driving environment of the vehicle 100.

[0167] In step S703, the identification processing unit 321 tracks objects whose identification numbers have been registered. The identification processing unit 321 identifies objects whose identification numbers have been registered among the objects detected in step S702. For example, an object whose identification number has been registered is an object that was detected and assigned an identification number in a previous object detection process.

[0168] If an object with an identification number already registered is detected, the identification processing unit 321 maps the object type and region-related information obtained in step S702 to the external information corresponding to the identification number (updating external information).

[0169] If it is determined that the object whose identification number has been registered does not exist in the image information, it is determined that the object associated with the identification number has moved outside the shooting angle of the camera device 110 (lost), and tracking is stopped.

[0170] In step S704, the identification processing unit 321 determines whether each object detected in step S702 is a new object whose identification number has not been registered. Furthermore, a new identification number is assigned to external information indicating the type and region of the object identified as a new object, and registered in the external information.

[0171] In step S705, the identification processing unit 321 outputs the generated external information along with the time information to the path generation device 150. Thus, the external information generation process performed by the identification processing unit 321 ends. However, this process is repeated periodically. Figure 7A The process continues until the user issues an end instruction (not shown).

[0172] Figure 7BThis is a flowchart illustrating an example of the processing performed by the distance information generation unit 322 to generate distance information for each object. The distance information generation unit 322 generates object distance information representing the distance values ​​of each detected object based on external information and distance image data.

[0173] Figure 8C This is an example of mapping distance image data to... Figure 8A An example diagram of the distance image 820 generated from image 810 based on image data. Figure 8C In the distance image 820, distance information is represented by color intensity, where darker colors indicate closer distances and lighter colors indicate farther distances.

[0174] In step S711, the distance information generation unit 322 counts the number N of objects detected by the identification processing unit 321 and calculates the number of detected objects Nmax, which is the total number of detected objects.

[0175] In step S712, the distance information generation unit 322 sets N to 1 (initialization process). The processing in step S713 and subsequent processing are performed sequentially for each object represented in the external information. It is assumed that the processing from step S713 to step S716 is performed sequentially starting from the object with the smaller identification number in the external information.

[0176] In step S713, the distance information generation unit 322 identifies a rectangular region in the distance image 820 corresponding to the region (object frame) in the image 810 for the Nth object included in the external information. The distance information generation unit 322 sets a frame (object frame) representing the outer shape of the corresponding region in the distance image 820.

[0177] Figure 8D This is a schematic diagram showing a bounding box representing the outer shape of the region set in image 820, superimposed on each object detected from image 810. (Example) Figure 8D As shown, the distance information generation unit 322 sets an object frame 821 corresponding to the person 801, an object frame 822 corresponding to the vehicle 802, and an object frame 823 corresponding to the sign 803 in the distance image 820.

[0178] In step S714, the distance information generation unit 322 generates a frequency distribution of distance information for pixels included in the rectangular region of the distance image 820 corresponding to the Nth object. If the information of each pixel mapped to the distance image data is a distance value D, then the intervals of the frequency distribution are set such that the reciprocals of the distances are evenly spaced.

[0179] If the defocus or disparity is mapped to each pixel of the distance image data, it is desirable to divide the frequency distribution intervals at equal intervals. In step S715, the distance information generation unit 322 uses the distance information that appears most frequently in the frequency distribution as the object distance information representing the distance of the Nth object.

[0180] The average distance values ​​within a region can be calculated as the object's distance information. When calculating the average, a weighted average based on reliable data can be used. By assigning greater weights to each pixel when the distance value has higher reliability, the object's distance value can be calculated more accurately.

[0181] The desired object distance information is information representing the distance from a predetermined position in the vehicle 100 to the object, in order to facilitate path generation in the path generation process described below. If the distance value D is used as the distance information, since the distance value D represents the distance from the camera element 302 to the object, the most frequently occurring value can be offset by a predetermined amount and can be set to represent the distance from a predetermined position (e.g., the front end) in the vehicle 100 to the object.

[0182] If the defocus amount ΔL is used as distance information, after converting to distance relative to the imaging element 302 using mathematical expression 6, the defocus amount ΔL is offset by a predetermined amount and set to represent information indicating the distance from a predetermined position in the vehicle 100 to the object.

[0183] In step S716, the distance information generation unit 322 determines whether N+1 is greater than the number of detected objects Nmax. If N+1 is less than the number of detected objects Nmax (in the case of "no" in step S716), then in step S717, the distance information generation unit 322 sets N to N+1, and the process returns to step S713.

[0184] In other words, object distance information is extracted for the next object (the (N+1)th object). In step S716, if N+1 is greater than the number of detected objects Nmax (if "yes" is true in step S716), the process proceeds to step S718.

[0185] In step S718, the distance information generation unit 322 outputs the object distance information for each of the Nmax objects, along with time-related information, to the path generation unit 330, and the processing ends. This information is stored in the memory 180. Additionally, the distance information relative to the radar device 120, along with the time information, is also stored in the memory 180. This process is repeated periodically. Figure 7B The process continues until the user issues an end instruction (not shown).

[0186] Object distance information is generated for each object included in the external information through object distance information generation processing. In particular, by statistically determining the object distance information based on the distance information included in the region of distance image 820 corresponding to the object detected in image 810, non-uniformity of distance information for each pixel caused by noise or calculation accuracy can be suppressed.

[0187] Therefore, information representing the distance to an object can be obtained with greater accuracy. As mentioned above, the methods used to statistically determine distance information are those that obtain the most frequently occurring distance information, the average or median, etc., in the distribution of distance information, and various methods can be employed.

[0188] Next, the processing of generating path information (path generation processing) performed by the path generation unit 330 of the path generation ECU 130 will be described. The path information includes the vehicle's direction of travel and speed. The path information can also be referred to as operational plan information. The path generation unit 330 outputs the path information to the vehicle control ECU 140. Based on the path information, the vehicle control ECU 140 controls the vehicle's direction of travel or speed via the control drive unit 170.

[0189] In this embodiment, if there are different vehicles (the preceding vehicles) in the driving direction of the vehicle 100, the path generation unit 330 generates path information for the following vehicle. Furthermore, it is assumed that the path generation unit 330 generates path information that allows the vehicle 100 to avoid collisions with objects by taking avoidance actions.

[0190] Figure 9 This is a flowchart illustrating an example of the path generation process performed by the path generation unit 330 according to the first embodiment. The process is performed by causing the computer included in the path generation apparatus 150 to execute a computer program stored in its memory. Figure 9 The flowchart shows the steps involved.

[0191] For example, if an instruction to start path generation processing is issued via user input, path generation unit 330 will begin. Figure 9 The path generation ECU 130 processes the path information of the vehicle 100 based on external information, object distance information, and distance information generated by the radar device 120. The path generation ECU 130 processes the path information by, for example, reading external information, object distance information, and distance information generated by the radar device 120 at various times from the memory 180 included in the path generation device 150.

[0192] In step S901, the path generation unit 330 detects objects on the expected travel path of the vehicle 100 based on external information and object distance information. The path generation unit 330 determines the objects on the travel path by comparing the expected orientation of the vehicle 100 in the travel direction with the positions and types of objects included in the external information.

[0193] The expected driving direction of the vehicle 100 is identified based on information related to the control of the vehicle 100 (steering angle, speed, etc.) obtained from the vehicle control ECU 140. If no object is detected on the driving path, the path generation unit 330 determines that "no object exists".

[0194] For example, suppose the camera device 110 acquires Figure 8A Image 810 is shown. If the path generation unit 330 determines, based on information related to the control of the vehicle 100 obtained from the vehicle control ECU 140, that the vehicle 100 is moving forward in the direction of the traffic lane 805, then the path generation unit 330 detects the vehicle 802 as an object on the travel path.

[0195] In steps S902 and S903, the path generation unit 330 determines whether to generate path information for following or for avoiding actions based on the distance between the vehicle 100 and the object on the driving path and the speed Vc of the vehicle 100.

[0196] In step S902, the path generation unit 330 determines whether the distance between the vehicle 100 and an object on the travel path is shorter than a threshold Dth. The threshold Dth is represented by a function of the travel speed Vc of the vehicle 100. The higher the travel speed, the larger the threshold Dth.

[0197] If the path generation unit 330 determines that the distance between the vehicle 100 and the object on the travel path is shorter than the threshold Dth ("Yes" in step S902), the process proceeds to step S903. If the path generation unit 330 determines that the distance between the vehicle 100 and the object on the travel path is equal to or longer than the threshold Dth ("No" in step S902), the process proceeds to step S908.

[0198] In step S903, the path generation unit 330 determines whether the relative speed between the vehicle 100 and the object on the travel path is positive. The path generation unit 330 obtains the identification number of the object on the travel path from external information, and obtains the object distance information of the object on the travel path at each time from the external information obtained during the time period from the current time to the predetermined time.

[0199] The path generation unit 330 calculates the relative speed between the vehicle 100 and the objects on the travel path based on the object distance information of the objects on the travel path during the time period up to a predetermined time. If the relative speed (speed of the vehicle 100 - speed of the objects on the travel path) has a positive value, it means that the vehicle 100 and the objects on the travel path are close to each other.

[0200] If the path generation unit 330 determines that the relative speed between the vehicle 100 and the object on the travel path is positive (if "yes" is true in step S903), the process proceeds to step S904. If the path generation unit 330 determines that the relative speed between the vehicle 100 and the object on the travel path is not positive (if "no" is true in step S903), the process proceeds to step S908.

[0201] Here, if the process proceeds to step S904, path information for performing avoidance actions is generated. Alternatively, if the process proceeds to step S908, path information for performing follow-along driving while maintaining a distance relative to the preceding vehicle is generated.

[0202] In other words, if the distance between the vehicle 100 and an object on the travel path is less than a threshold Dth and the relative speed between the vehicle 100 and the object on the travel path is positive, the path generation unit 330 performs an avoidance action. On the other hand, if the distance between the vehicle 100 and an object on the travel path is equal to or greater than the threshold Dth, the path generation unit 330 determines that it will follow the object.

[0203] Alternatively, if the distance between the vehicle 100 and the object on the travel path is less than a threshold Dth, and the relative speed between the vehicle 100 and the object on the travel path is zero or has a negative value (if they are moving away from each other), the path generation unit 330 determines to follow.

[0204] If the distance between the vehicle 100 and an object on the travel path is shorter than a threshold Dth obtained based on the speed of the vehicle 100, and the relative speed between the vehicle 100 and the object on the travel path is positive, then it is conceivable that the vehicle 100 is likely to collide with the object on the travel path.

[0205] Therefore, the path generation unit 330 generates path information for avoiding obstacles. Otherwise, the path generation unit 330 follows the path. The aforementioned determination may include determining whether the detected object on the driving path is a mobile device (such as a car or motorcycle).

[0206] In step S904, the path generation unit 330 begins processing to generate path information for performing avoidance behavior.

[0207] In step S905, the path generation unit 330 acquires information related to the avoidance space. The path generation unit 330 acquires distance information including prediction information, which includes the distance from the radar device 120 to an object on the side or rear of the vehicle 100, or the distance to the object.

[0208] The path generation unit 330 obtains information indicating the direction and range of the space in which the vehicle 100 can move around the vehicle 100 based on the distance information (fourth distance information) obtained from the radar device 120 and information indicating the speed and size of the vehicle 100.

[0209] In this embodiment, the distance information (fourth distance information) obtained from the radar device 120 is used to avoid space, but the distance information can be used to generate integrated distance information about the object.

[0210] In step S906, the path generation unit 330 sets path information for the avoidance behavior based on information indicating the direction and range of the space that the vehicle 100 can move to, external information, and object distance information. For example, the path information for the avoidance behavior is information used to change the route of the vehicle 100 to the right when there is space on the right that the vehicle 100 can avoid, for example, while reducing speed.

[0211] In step S907, the path generation unit 330 outputs path information to the vehicle control ECU 140. The vehicle control ECU 140 determines the parameters for controlling the drive unit 170 based on the path information and controls the drive unit 170 so that the vehicle 100 travels along the path represented by the acquired path information.

[0212] Specifically, the vehicle control ECU 140 determines the steering angle, accelerator control value, brake control value, control signal for shifting gears, and light illumination control signal based on path information.

[0213] Here, step S907 is used as a path generation step (path generation unit) to generate path information based on distance information.

[0214] On the other hand, in step S908, the path generation unit 330 begins processing to generate path information for performing the follow operation.

[0215] In step S909, the path generation unit 330 generates path information used by the vehicle 100 to follow the object (the preceding vehicle) on the driving path. Specifically, the path generation unit 330 generates path information such that the distance between the vehicle 100 and the preceding vehicle (vehicle-to-vehicle distance) is maintained within a predetermined range.

[0216] For example, if the relative speed between the vehicle 100 and the preceding vehicle is zero or negative, or if the distance between vehicles is equal to or greater than a predetermined distance, the path generation unit 330 generates path information such that while the vehicle 100 maintains a straight-line direction of travel, the predetermined distance between vehicles is maintained by accelerating and decelerating.

[0217] The path generation unit 330 generates path information such that the vehicle 100's speed does not exceed a predetermined value (e.g., the legal speed limit of the road on which the vehicle 100 is traveling or a set speed based on instructions from the driver 101). After step S909, the process proceeds to step S907, and the vehicle control ECU 140 controls the drive unit 170 based on the generated path information.

[0218] Next, in step S910, it is determined whether the user has issued an instruction to end the path generation process. If "yes", the path generation unit 330 ends the process of generating path information. If "no", the process returns to step S901, and the process of generating path information is repeated.

[0219] Based on the above control, statistical processing of distance information within a frame based on the position and size of objects in the image can reduce the influence of sensor noise or local distance errors caused by high brightness reflections from the subject, and can calculate the distance value relative to each object with high accuracy.

[0220] Furthermore, since the distance values ​​relative to each object can be highly accurate and the path of the vehicle 100 calculated by the path generation ECU 130 can be accurately calculated, the vehicle 100 can drive more stably.

[0221] <Variation Example 1>

[0222] In the above processing, the region of an object is represented as a rectangular region (object box) including the object, but the region of an object can also be a region with the shape of the object, having the outer perimeter of the object within the image as its boundary. In this case, in step S702, the identification processing unit 321 stores the region in image 810 containing the object as the object region in external information. For example, the identification processing unit 321 can segment the region for each object by recognizing the attributes of each pixel in the image data.

[0223] Figure 8E This is a schematic diagram illustrating an example of the identification processing unit 321 segmenting regions for each object and overlaying the results onto image 810. Region 831 represents the region of person 801, region 832 represents the region of vehicle 802, and region 833 represents the region of sign 803. Furthermore, region 834 represents the region of road 804, and region 835 represents the region of lane 805.

[0224] In this case, in steps S713 and S714, the distance information generation unit 322 only needs to perform the following steps for each object: Figure 8E The regions shown are used to calculate, for example, the frequency distribution of distance values ​​included within those regions.

[0225] By defining the object's region in this way, distance information from the background and other elements besides the object is less likely to be included within the region. In other words, the distribution of distance information within the region can better reflect the object's distance. Therefore, since the influence of areas other than the object, such as the object's background or foreground, can be reduced, the object's distance value can be calculated more accurately.

[0226] <Variation Example 2>

[0227] Image information and distance image information are sequentially output from the camera device 110 of this embodiment. Furthermore, the identification processing unit 321 uses the sequentially received image information to sequentially generate external information. The external information includes an object identification number, and if an object with the same identification number is detected at any of the times T0 and T1, the time-varying changes in the object's distance information or the detected size can be determined.

[0228] Therefore, in Modification 2, the distance information generation unit 322 calculates the average distance value D of objects with the same identification number within a predetermined time range. Thus, the non-uniformity of distance in the time direction is reduced.

[0229] Figure 10This is a flowchart illustrating an example of the object distance information generation process performed by the distance information generation unit 322 according to Modified Example 2. The process is performed by causing the computer included in the path generation apparatus 150 to execute a computer program stored in its memory. Figure 10 The flowchart outlines the operations for each step. Regarding the processes within this flowchart, due to... Figure 7B The processes shown have the same numbering, indicating the processes related to... Figure 7B The process described is the same, so its explanation will be omitted.

[0230] In step S1000, the distance information generation unit 322 uses the most frequently occurring distance information in the frequency distribution generated in step S714 as the object distance information representing the distance to the Nth object. Furthermore, the distance information generation unit 322 saves (stores) the object distance information along with the identification number and time in the memory 340.

[0231] In step S1001, the distance information generation unit 322 retrieves the historical object distance information stored in the memory 340 that has the same identification number as the Nth object. The distance information generation unit 322 also retrieves object distance information with the same identification number corresponding to the time period from the moment corresponding to the latest object distance information until a predetermined time has elapsed.

[0232] Figure 11 This is an illustrative diagram illustrating the change over time of object distance information for an object with the same identification number as the Nth object. The horizontal axis represents time, and the vertical axis represents the object distance information (distance value D). Time t0 represents the time when the latest distance value D was obtained.

[0233] In step S1002, the distance information generation unit 322 calculates the average value of object distance information within a time range from the moment the latest object distance information is acquired until a predetermined time, based on the history of object distance information acquired from objects with the same identification number as the Nth object. For example, in Figure 11 In the process, the distance information generation unit 322 calculates the average distance values ​​at four points included in the predetermined time range ΔT.

[0234] As described above, by using the identification number included in the external information to obtain the historical distance information (distance value) of the same object and calculating the time average, non-uniformity can be suppressed. Even if the road traveled by the vehicle 100 changes (e.g., curves, slopes, or rugged paths), the average value in the time direction can be calculated while tracking the same object.

[0235] Therefore, while reducing the impact of changes in the driving environment, it is possible to reduce the non-uniformity of distance values ​​caused by noise such as optical shot noise included in image signals, and to calculate the object's distance value more accurately. In Modification 2, to achieve a similar effect, object distance information that has undergone a certain degree of time averaging can be obtained through a low-pass filter when acquiring object distance information.

[0236] <Variation Example 3>

[0237] In the above-described variation 2, non-uniformity is suppressed by performing a time-based averaging of object distance information with the same identification number. When averaging the history of object distance information over a predetermined time range, the number of samples used for averaging can be increased by employing a longer time range. For this reason, the non-uniformity of the object distance values ​​relative to the vehicle 100 can be further reduced.

[0238] However, if the distance from the vehicle 100 to the object changes within a predetermined time range, there is a possibility that the distance value of the object relative to the vehicle 100 may not be accurately estimated because averaging is performed while including the change in distance. In Modification 3, by using a weighted average of the object distance information using the sizes of objects with the same identification number, the distance between the vehicle 100 and the object can be obtained with higher accuracy.

[0239] Figure 12 This is a flowchart illustrating an example of the object distance information generation process performed by the distance information generation unit 322 according to Modified Example 3. The process is performed by causing the computer included in the path generation apparatus 150 to execute a computer program stored in its memory. Figure 12 The flowchart outlines the operations for each step. Regarding the processes within this flowchart, due to their relationship with… Figure 7B and Figure 10 The processes with the same number shown are the same as those described above, so their descriptions will be omitted.

[0240] In step S1000, the distance information generation unit 322 uses the distance information that appears most frequently in the frequency distribution as the object distance information representing the distance to the Nth object. Furthermore, the distance information generation unit 322 stores the object distance information along with the identification number and time in the memory 340.

[0241] In step S1201, the distance information generation unit 322 retrieves from the memory 340 the history of object distance information for objects with the same identification number as the Nth object, as well as the history of information representing the size of the object. The information representing the size of the object is obtained from the information representing the object frame stored in the external information. For example, the width (x1-x0) based on the coordinates (x0, y0) of the upper left side and the coordinates (x1, y1) of the lower right side is used as the information representing the size of the object.

[0242] In step S1202, the distance information generation unit 322 uses information representing the size of objects with the same identification number as the Nth object to perform a weighted average of the distance information of objects with the same identification number corresponding to the time period from the time corresponding to the latest object distance information until the time before a predetermined time. The weight coefficient at each time point is determined using the object size at the corresponding time point.

[0243] Figures 13A to 13D This is an illustrative diagram of a weighted average. Figure 13A This is a schematic diagram illustrating an image 1300 based on image data acquired by the camera device 110 at time t1, prior to time t0 corresponding to the latest object distance information. Image 1300 includes a vehicle 1301. Box 1311 represents the object frame of the vehicle 1301 as determined by image 1300.

[0244] Figure 13B This is a schematic diagram illustrating image 1310 based on image data acquired by camera device 110 at time t0. Similar to image 1300, image 1310 includes a vehicle 1301. Additionally, box 1312 represents an object frame in image 1310 corresponding to the vehicle 1301. At time t0, the size of the vehicle 1301 in image 1300 is larger than the size of image 1310 acquired at time t1. Object frame 1312 is larger than object frame 1311.

[0245] Since the size of an object within an image is proportional to the lateral magnification of the camera optical system 301, the distance between the object and the vehicle 100 is proportional to the reciprocal of the object's size in the image information. By comparing the object's size in the image information at different times, for example, if the size increases, it can be determined that the distance between the object and the vehicle 100 has decreased, and conversely, if the size decreases, it can be determined that the distance between the object and the vehicle 100 has increased. Furthermore, if the change in size is small, it can be determined that the change in distance between the object and the vehicle 100 is small.

[0246] In the following text, as an example, it is assumed that distance information is obtained when vehicle 1301 is the Nth object. Figure 13C This is a schematic diagram illustrating the change over time of object distance information for an object having the same identification number as vehicle 1301 (the Nth object). Figure 13D This is a schematic diagram illustrating the reciprocal of the information representing the size (width) of an object having the same identification number as vehicle 1301 (the Nth object) over time.

[0247] In step S1202, the distance information generation unit 322 compares the reciprocal of the size (width) of the object at each time point from time t0 until a predetermined time with the reciprocal of the size (width) of the object at time t0. The distance information generation unit 322 determines a weighting coefficient such that the weighting coefficient decreases as the absolute value of the difference between the reciprocals of the object's size (width) increases.

[0248] The relationship between the reciprocal of the object's size (width) and the weighting coefficient is not limited to the example above. For instance, the weighting coefficient can be determined based on the ratio of the reciprocal of the object's size (width) at each time point to the reciprocal of the object's size (width) at time t0.

[0249] In step S1202, the distance information generation unit 322 uses weighting coefficients to perform a weighted average of the object distance information to obtain the object distance information of the vehicle 1301 at time t0. According to the processing in Modified Example 3, by using weighting coefficients determined by the size of the object in the image to perform a weighted average of the object distance information, the estimation error of the distance value caused by the change in the relative distance from the vehicle 100 to the object can be reduced.

[0250] <Second Embodiment>

[0251] In the first embodiment, by statistically processing the distance data obtained from the camera phase difference system for the object, the distance to the object included in the image (object distance information) is obtained with higher accuracy.

[0252] In the second embodiment, by combining the method for calculating distance using a camera phase difference system with the method for calculating distance using image recognition, the distance to an object can be obtained with higher accuracy. Hereinafter, the method for calculating distance and the processing based on that method will be referred to as "distance measurement".

[0253] The second embodiment of the present invention will be described in detail below with reference to the accompanying drawings. The following processing can be performed by any or a combination thereof of the processor constituting the image processing unit 310 and the object information generation unit 320 of the imaging device 110 and the path generation ECU 130.

[0254] In the second embodiment, the path generation device 150 corrects the ranging value by combining the camera phase difference ranging and the ranging using image recognition.

[0255] Camera plane phase difference ranging is ranging using the camera plane phase difference system described in the first embodiment. Image recognition-based ranging is ranging that calculates the distance based on the width of an object detected by object recognition (object width ranging), or ranging that calculates the distance based on information about the object's grounding position (grounding position ranging).

[0256] In object width ranging, the distance to an object is calculated by utilizing the fact that the object is at a long distance when the number of pixels representing its width in the image decreases, and at a short distance when the number of pixels representing its width in the image increases. Other parameters representing the size of an object in the image (e.g., the size of the object box) such as height or tilt direction can be used in a similar manner in the distance calculation.

[0257] In grounding distance measurement, it is assumed that the object is grounded, for example, to the road surface, and the distance from the vehicle 100 to the object is calculated based on the grounding wire of the object in the image (which becomes the lower end in the case of an image where the road surface is on the lower side) and the distance to the vanishing point in the image. The closer the grounding wire is to the vanishing point, the longer the distance from the object to the vehicle 100; conversely, the farther the grounding wire is from the vanishing point, the shorter the distance from the object to the vehicle 100.

[0258] This section will explain the characteristics of errors in camera phase difference ranging, object width ranging, and ground position ranging. Regarding the consideration of common errors, relative error and absolute error are defined as follows: If there is no variation in relative distance, the relative error is defined as the quantity corresponding to the standard deviation relative to a sufficient sample. If there is no variation in relative distance, the absolute error is defined as the quantity corresponding to the difference between the mean and the true value relative to a sufficient sample.

[0259] The primary cause of relative error in camera phase difference ranging is disparity error in block matching caused by pixel value inhomogeneity due to sensor noise. Since the relative error does not change with the disparity value, it deteriorates substantially proportionally to the square of the distance when converted to distance.

[0260] Absolute error arises from aberrations in the optical system, assembly errors, and variations caused by heat or vibration. Corrections can be made for each cause; however, if computational costs are not accounted for, large absolute errors may remain.

[0261] The relative error in object width ranging depends on the resolution and recognition accuracy of the object in the image. In object width ranging, the width of the detected object in the image cannot be converted into a distance unless the actual object width (the actual width of the object on the object side, i.e., a physical quantity expressed in meters, etc.) is known.

[0262] For this reason, the actual object width needs to be determined in any way possible, such that both the absolute and relative errors depend on the actual object width. Since the relative error is proportional to the distance, there is a possibility that the relative error will be smaller compared to phase difference ranging over long distances.

[0263] The relative error in grounding distance measurement depends on the accuracy of grounding wire identification of the object in the image and the image resolution. Since the image resolution becomes the resolving power used to measure the distance between the vanishing point and the grounding wire, high-resolution images can be used for high-accuracy measurements even at long distances. Furthermore, if the road surface extends downwards, the estimation error of the pitch angle along the optical axis becomes the ranging error.

[0264] If a camera device for acquiring images is installed in a mobile device, the pitch angle will vary for each camera frame due to movement acceleration or road conditions. In this case, the pitch angle error becomes a relative error. An error in the pitch angle that is always a constant value due to the installation state or the tilt of the mobile device itself becomes an absolute error. Although this will be explained below, using information such as the vanishing point and movement information to estimate the pitch angle can reduce the error.

[0265] Although the relative error is proportional to the distance and equal to the object's width, the error is greater than the object's width because the estimation of the pitch angle is added to the relative error. Since distance measurement based on the object's grounding location requires contact with the ground, there is a problem that distance measurement cannot be performed if the lower end is not in contact with the ground (such as with a signal or marker).

[0266] Figure 14 This is a flowchart illustrating an example of a process for obtaining ranging values ​​(object distance information) according to the second embodiment. This is performed by causing the computer included in the path generation apparatus 150 to execute a computer program stored in its memory. Figure 14 The flowchart shows the steps involved.

[0267] Data D1401 is the data input to the distance information generation unit 322. As described in the first embodiment, external information (object identification number, object type (attribute), and the size of the area corresponding to the object) related to the object of the image obtained by the imaging element 302 through image recognition is input to the distance information generation unit 322.

[0268] Data D1401 may include the results of other image recognition processes and may include information identifying the locations of ground pixels in the image, such as image coordinates identifying the image range of the identified image or information about the same object region using semantic region segmentation techniques. Additionally, data D1401 may include other information related to the image recognition results.

[0269] Additionally, distance image data representing the result (distance information of each pixel) obtained through calculations by the camera phase difference system is input as data D1401 to the distance information generation unit 322. In this embodiment, in addition to this data, the focal length f of the camera optical system 301, information related to the moving speed of the vehicle 100, and information related to the installation position of the camera device 110 are also input as data D1401 to the distance information generation unit 322. Data D1401 is a data set that integrates this information.

[0270] In the following steps, identification processing can be performed on several objects simultaneously. However, while processing is carried out while preserving and referencing time-series data, processing is performed only on objects that have already been identified as the same object. That is, objects with the same identification number are used as input data.

[0271] In step S1401, the distance information generation unit 322 obtains the distance value (distance value) D1 through ground position ranging and outputs it as data D1402. The distance value D1 represents the distance between the vehicle 100 (camera device 110) and the target object calculated by ground position ranging.

[0272] Data D1402 represents the distance value D1 (second distance information) calculated by distance measurement at the grounding position in step S1401. Here, step S1401 is used as a second distance information acquisition step (second distance information acquisition unit), which acquires the second distance information based on the end position information of the object included in the image signal, wherein the end position includes the grounding position or the bottom position.

[0273] The distance information generation unit 322 uses image coordinates and other information, such as the image range of the identified object included in the data D1401, to obtain the pixel positions in the image that are in contact with the ground. An overview of the distance measurement process will be given in the following case: the optical axis is set at a height H parallel to the road surface, and the vanishing point in the image is separated from the ground line by Hs pixels (or sub-pixel units).

[0274] When the pixel size in the image obtained at focal length f by the central projection method (which can be an image corrected by the central projection method) is Ps, the following expression 7 can be used to represent the distance value D1.

[0275]

[0276] Even if not on the road surface, if the optical axis is not parallel to the ground plane, if the central projection method is not used, or if there is significant distortion aberration, the distance measurement itself can still be performed, as long as the vanishing point and ground plane can be estimated.

[0277] In mathematical expression 7, although it is assumed that the optical axis is parallel to the road surface, if an error exists in the pitch angle of the mobile device as described above, the vanishing point position will become a different position than the assumed position. Therefore, an error occurs in the value of Hs, leading to distance error. Furthermore, in cases of poor identification accuracy, a similar distance error occurs because Hs is identified as a position different from the actual grounding wire.

[0278] In step S1402, the distance information generation unit 322 obtains the distance value (distance value) D2 by camera surface phase difference ranging and outputs it as data D1404. Data D1404 represents the distance value D2 (first distance information) calculated by camera surface phase difference ranging in step S1402.

[0279] In other words, the first distance information is the distance information obtained by using a phase difference ranging method based on the signals from the first photoelectric conversion unit and the second photoelectric conversion unit described above. Alternatively, the first distance information is the distance information obtained by using a phase difference ranging method based on the two image signals from the stereo camera. Here, step S1402 is used as a first distance information acquisition step (first distance information acquisition unit) for acquiring first distance information corresponding to the object included in the image signal.

[0280] As described in the first embodiment, the distance information generation unit 322 can obtain the distance information (distance value) of the target object based on distance image data and external information. For example, regarding the input data D1401, it is assumed that the input is distance image data in which the distance information of each pixel is represented by the defocus amount and external information of the region of the object represented by the bounding box.

[0281] At this time, the distance information generation unit 322 can use the most frequently occurring defocus value included in the object frame of the target object and the focal length f to obtain the distance from the vehicle 100 to the target object (object distance information) according to the imaging expression. The obtained distance is acquired as the ranging value D2. The distance value itself can be input as data D1401, and other data during the calculation can also be used.

[0282] In step S1403, the distance information generation unit 322 acquires the width Ws (object width) of the target object in the image. The object width Ws is represented by the number of pixels. The number of objects can be represented by sub-pixel units. The distance information generation unit 322 measures the object width Ws based on the image recognition result.

[0283] For example, if the external information included in data D1401 includes information about the object frames representing each object within the image, then the width of the object frame corresponding to the target object can be used as the object width Ws. Alternatively, it doesn't have to be the object's width; it can also be its height. Any object frame with a larger pixel count can be selected, or both can be used to enhance robustness.

[0284] If the object's attribute information is known, it can be considered when making the determination. Additionally, for example, if the object is a vehicle located off-center from the field angle, the object's width is likely to include the sides of the vehicle. In this case, the height is the better choice.

[0285] If changes in the time series are observed, considering changes in the ratio between height and width, information that is more stable than distance changes can be selected. The distance information generation unit 322 uses information representing the object width Ws as data D1403.

[0286] In step S1404, the distance information generation unit 322 uses the object width Ws from data D1403, and any one or two of the ranging value D1 obtained by ranging from the ground position measurement in data D1402 and the ranging value D2 obtained by ranging from the camera surface phase difference measurement in data D1404. Furthermore, it calculates the actual object width W. The actual object width W is information expressed using a unit system (meter, etc.) that represents the width of the target object using length.

[0287] As described above, the actual object width W can be obtained using one or both of the measurement values ​​D1 and D2, relative to the object width Ws. However, it is desirable to choose the measurement value with the smaller absolute error between D1 and D2. If the measurement value D1 is used, the actual object width W can be represented as follows using the expression 8.

[0288]

[0289] Furthermore, the actual object width W can be determined based on information about the types of objects included in the external information representing the input data of D1401. That is, integrated distance information can also be generated based on the type of object. For example, if the target object is a passenger vehicle, the actual object width W can also have a preset value (e.g., 1.7m).

[0290] However, if the actual object width W is determined based on the type of object, then since the actual object width W is strictly different for each object, this difference becomes an absolute error. The distance information generation unit 322 outputs information representing the actual object width W as data D1405.

[0291] In step S1405, the distance information generation unit 322 uses the actual object width W and the object width Ws to obtain the distance value D3. Data D1406 (third distance information) represents the distance value D3 calculated using the object width in step S1405.

[0292] Here, step S1405 serves as a third distance information acquisition step (third distance information acquisition unit), which acquires third distance information based on the size (width or height) of the object included in the image signal. The processing performed in step S1405 becomes the inverse processing of the processing in step S1404. The ranging value (distance value) D3 can be represented as follows using the following expression 9.

[0293]

[0294] Here, since the actual object widths W are the same, D1 = D3 holds true. However, since the time-series information, which will be explained below, is used to add processing to step S1404, D1 and D3 become different distance values.

[0295] The step group C1101, consisting of steps S1403, S1404, and S1405, represents the object width range. In step S1406, the distance information generation unit 322 integrates the distance measurement value D1 of data D1402, the distance measurement value D3 of data D1406, and the distance measurement value D2 of data D1404, and uses it as the distance value D relative to the identified object.

[0296] It is not necessary to combine all the first to third distance information. Integrated distance information, which combines and integrates at least two of the first to third distance information, can be generated. In this embodiment, both a second distance information acquisition unit and a third distance information acquisition unit are provided, but at least one of these two units can be provided.

[0297] Here, step S1406 is used as a distance information integration step (distance information integration unit), which generates integrated distance information that combines and integrates at least two of the first distance information to the third distance information.

[0298] For example, the integration process only requires selecting distance measurement value D1 (first distance information), distance measurement value D2 (second distance information), and distance measurement value D3 (third distance information) as distance measurement value D.

[0299] Distance values ​​D1, D2, and D3 each have different relative and absolute errors depending on the type of distance measurement acquired. The distance information generation unit 322 can select distance values ​​with smaller relative and absolute errors from among multiple distance measurement methods, depending on the scenario.

[0300] For example, regarding the ranging values ​​obtained by grounding position ranging or camera phase difference ranging, larger distance values ​​have smaller errors. Therefore, if the obtained ranging value is greater than (longer than) a predetermined distance, the distance information generation unit 322 can select either ranging value D1 or ranging value D2, and if the obtained ranging value is equal to or less than (shorter than) a predetermined distance, it can select ranging value D3.

[0301] Integrated distance information can be generated by weighted summation of at least two of the first to third distance information.

[0302] Furthermore, regarding another integrated processing technique, by calculating the distribution of the existence probability relative to distance by taking into account each of the absolute error and the relative error, a selection can be made from the sum of probability distributions to maximize the existence probability. Additionally, the existence probability of the relative distance value can be determined relative to the current relative distance value based on information included in data D1401 regarding the vehicle's speed, accelerator, brakes, and steering relative to the vehicle 100.

[0303] For example, it can be determined that when the probability of having the same acceleration relative to the previous moment is the highest, the probability decreases when the acceleration change increases, thus preventing the acceleration change from increasing. Based on this, the probability of the existence of relative distance can be calculated.

[0304] Furthermore, if accelerator information is available, the maximum probability can be determined in the direction of increasing acceleration, and if braking information is available, the maximum probability can be determined in the direction of decreasing acceleration. Additionally, settings can be configured based on the type of target object. If the target object is a car or an autonomous two-wheeled vehicle, the probability of a significant change in relative distance increases because there is a possibility that the identified object will accelerate or decelerate.

[0305] On the other hand, if the category is pedestrians, etc., who do not suddenly accelerate or decelerate, then the change in relative distance will depend on the probability of their own actions, thus allowing the probability of existence to be determined with higher accuracy. The basic process of inputting data at a certain moment has been explained above. Subsequently, the process of inputting continuous data in a time series will be described.

[0306] Figure 15 This is a flowchart illustrating a portion of a processing example of multiple consecutive input data in a time series according to the second embodiment, and Figure 16 It is shown Figure 15 The flowchart shows the rest of the processing example.

[0307] By causing the computer program stored in the computer execution memory included in the path generation device 150 to perform... Figure 15 and Figure 16 The flowchart illustrates the operations for each step. Apply the same reference numerals to the steps described above. Figure 14 The processing and data described herein are the same, and their descriptions will be omitted.

[0308] If the input data is continuous in a time series, then for objects assigned the same identification number (that is, objects identified as the same object), the ranging value D1 of D1402, the object width Ws of D1403, and the ranging value D2 of D1404 can be obtained sequentially in a time series.

[0309] If continuous input data is given in a time series, the distance measurements will change over time due to the variation in the relative distance to the target object. However, if the object is assumed to be a rigid body, then the actual object width W can be said to remain constant.

[0310] For this reason, in step S1504, even though the relative distance value has changed over time, the distance information generation unit 322 obtains the average actual object width W' data D1505 by smoothing the actual object width W in the time series direction. Therefore, the relative error can be reduced.

[0311] At this point, it is best to remove the data that has become outliers, so that data with significant errors is not utilized. Furthermore, when smoothing is performed across all data in the time series direction, the effects of errors in the initial stage may remain even after time has elapsed.

[0312] In this situation, improvement can be achieved by calculating a moving average over a predetermined range. The number of data points used in the moving average can be determined by considering both relative and absolute errors. If the system has a significant relative error, it is best to use a large number of data points for the moving average; conversely, if the absolute error is significant, it is best to use a small number of data points.

[0313] If W is estimated using mathematical expression 8, the relative errors, including the ground position distance and the object width (number of pixels), can be sufficiently reduced by smoothing the time series direction. However, the absolute error retains a quantity corresponding to the ground position distance. A similar consideration can be made even when calculating the actual object width W' using the camera plane phase difference distance.

[0314] Data D1505 becomes the object width W' with a small relative error, and due to the distance conversion process in step S1405, the object width distance value D3, which is data D1406, can be obtained as a distance value that has only the relative error in the object width Ws and the absolute error corresponding to the ground position distance value.

[0315] On the other hand, Figure 16 In step S1607, the distance information generation unit 322 calculates the correction amount for the absolute error in the camera plane phase difference distance measurement value D2 based on the distance measurement values ​​D1 and D3. In the absolute error of the camera plane phase difference distance measurement value D2, the error with a constant value becomes the main component through defocus conversion, regardless of the distance. For this reason, the distance information generation unit 322 converts the camera plane phase difference distance measurement value D2 into a defocus amount based on the focal length and the imaging expression.

[0316] Similarly, the distance information generation unit 322 converts the object width distance value D3 into defocus amount using the same focal length and the same imaging expression. At this time, the object width distance value D3 can be converted into defocus amount using either the ground position distance value D1 or the relative distance value D.

[0317] The distance information generation unit 322 calculates the average value of the time series difference data by taking the difference between the defocus amount calculated from the phase difference distance value D2 of the camera surface and the defocus amount calculated from the object width distance value D3 at the same time.

[0318] If sufficient data can be used to calculate the average value, the obtained average value represents the difference between the absolute errors in the camera phase difference ranging value D2 and the object width ranging value D3. If this average value is selected in step S1609, then data D1608 is used as the absolute error correction value.

[0319] If the grounding location distance value D1 is used for correction, the correction is based on the absolute error in the grounding location distance value D1, and if the grounding location distance value D1 is used in step S1504, the result is the same absolute error. In this case, the influence of relative error is reduced by selecting a value with a smaller relative error.

[0320] Step S1609 is the absolute error correction value selection process, and the distance information generation unit 322 determines which result between the processing result of step S1607 and the processing result of step S1608 is selected as the absolute error correction value. Details will be explained below.

[0321] In step S1402, the distance information generation unit 322 calculates the distance value D2 by measuring the phase difference of the camera surface, and uses the absolute error correction value of D1608 to correct the defocus amount. Since the absolute error correction value represents the defocus amount offset, the absolute error in the distance value D2 of D1404 is corrected by subtracting the offset from the defocus amount calculated based on the input data.

[0322] The direct distance value can be corrected. In fact, the direct distance value has a shape that is adjusted to the absolute error in the data used for the difference in step S1607. If the defocus amount calculated from the object width distance value D3 is used as described above, it becomes the absolute error in the object width distance value D3.

[0323] The object width measurement value D3 depends on the measurement value used in step S1504. Therefore, if the actual object width W is calculated using the ground position measurement value D1 in step S1504, all the ground position measurement values ​​D1, the camera surface phase difference measurement value D2, and the object width measurement value D3 have the absolute error in the ground position measurement value D1.

[0324] Since the absolute error is consistent across the three ranging values, only the relative error needs to be considered when determining the probability distribution in the integrated ranging process of step S1406. Therefore, the relative ranging value D of data D1507 can be calculated more simply and stably.

[0325] Using the time-series data of the input object as described above, the relative distance value D can be calculated in time series based on the grounding position distance value D1, the camera surface phase difference distance value D2, and the object width distance value D3. The relative velocity, relative acceleration, and relative jolt relative to the target object can be calculated based on the time-series data of the relative distance value D. This data can then be used to calculate the probability distribution of the aforementioned relative distance values.

[0326] For example, a probability distribution can be determined to reduce the change in relative acceleration. Furthermore, the calculation of the distance value D can be modeled by including the change in the relative distance value. In this case, the distance values ​​are represented in a linearly coupled manner by the following expression 10.

[0327] D = (1 - K1 - K2 - K3)D m +K1D1+K2D2+K3D3....(Expression 10)

[0328] Dm is a ranging value estimated from the dynamic model based on the relative velocity, relative acceleration, and relative jump relative to the target object, as described above. K1, K2, and K3 represent the weighting coefficients of D1, D2, and D3. That is, if K1, K2, and K3 can be estimated based on the probability distributions in each technique, the likelihood ranging value D can be obtained.

[0329] Assuming that the probability distributions of the various distance values ​​Dm, D1, D2, and D3 can be approximated as Gaussian distributions, the following expression 11 can be used to analytically solve for these probability distributions. Specifically, by taking the variances on both sides and performing partial differentials using the coefficient K, three expressions can be obtained for the three unknowns.

[0330]

[0331] V represents the variance of the probability distribution relative to each distance value Dx. X represents 1, 2, or 3. Therefore, K1, K2, and K3 can be calculated for the ranging values ​​Dm, D1, D2, and D3 obtained in each frame, and the ranging value D can be obtained. Each variance value can be preset in each ranging technique, or it can be calculated based on each ranging result or condition. For example, the grounding position ranging value D1 can be set considering the accuracy of identification in the grounding point calculation and the pitch angle estimation error of the camera instrument.

[0332] Similarly, the object width ranging value D3 also depends on the recognition accuracy in the image that determines the object width. Regarding the camera plane phase difference ranging value D2, if calculated using general block matching, there are many ranging points in the recognized image, and the variance can be calculated as its statistic.

[0333] At this point, since significant errors may still be included, it is best to remove outliers from the block matching results. Additionally, since the variance can be obtained as a statistic of the many ranging values ​​within the identified object, the variance depends on the number of ranging points; however, the block size of the block matching also needs to be considered.

[0334] This is because, when examining each ranging point, the blocks in a block-matching sequence overlap at adjacent ranging points, thus the ranging points are not independent as a result of noise affecting the sensor pixels. If the variance of the average phase difference ranging value D2 within the identified object is independent of the variance of each ranging point, then the former variance becomes the value obtained by dividing the variance of each ranging point by the number of ranging points.

[0335] However, since the variance is not independent for each ranging point as mentioned above, it is necessary to determine the degradation value by taking this into account. Specifically, this can be calculated as a product of the number of pixels within the block, or it can be a product of values ​​determined by some other method.

[0336] Alternatively, Vm can be updated using the results of the preceding frame. Specifically, since the posterior probability in the aforementioned model can be calculated, it can be updated by combining it with the posterior probability (primarily in the form of adding to the estimated variance). Furthermore, mathematical expressions for the three ranging values ​​have been described here. However, if other ranging techniques exist, Vm can also be increased, and even in this case, an analytical solution can be obtained using a similar concept to the expanded expression.

[0337] Conversely, it is not necessary to utilize all of D1, D2, and D3. For example, only one can be used depending on the situation. In particular, if the changes in velocity and acceleration, etc., are large, it is assumed that the error is large. Therefore, in such cases, it is conceivable not to utilize it.

[0338] For example, if the object ahead is confirmed to be stationary, an approximate change can be obtained from the speed of the host vehicle. Furthermore, since the variance of the ranging value D (the probability distribution under the assumption of a Gaussian distribution) can also be obtained through mathematical expression 11, there is a high probability of error in the ranging technique if the ranging values ​​D1, D2, and D3 deviate significantly. In this case, the coefficient K can be set to zero to be disregarded.

[0339] So far, time-series data has been described, but continuous acquisition is not always necessary. If object identification fails and the correct input data cannot be obtained for that frame, no processing is performed, or the calculation is restarted from there once the object can be correctly identified.

[0340] In this case, the average value W' of the actual object width or the absolute error correction value can be used as is, thereby obtaining a stable relative distance value D. However, in this case, since the probability distribution of the relative distance values ​​calculated based on the relative distance value D cannot be correctly obtained, it is desirable to set it to zero.

[0341] Additionally, cases where one or both of the following can not be correctly calculated: the grounding position distance value D1, the camera surface phase difference distance value D2, and the object width distance value D3. Examples include situations where the identified object is floating, and situations where different objects enter between the vehicle and the identified object, and the object width cannot be correctly measured.

[0342] In this situation, the change will be significantly different from previous values, significantly different from other distance values, and there is a high probability that it cannot be obtained correctly. In this case, the influence on the calculation of the relative distance value D can be avoided or reduced by setting zero or very small values ​​as having a probability distribution.

[0343] In step S15108, if the relative distance value D relative to the target object changes sufficiently with the time series, the distance information generation unit 322 performs a correction process to perform highly accurate absolute error correction using the change in relative distance and the change in the actual object width W.

[0344] Figure 17A and Figure 17B This is a schematic diagram illustrating an example of absolute error correction processing according to the second embodiment. A time axis is set for the horizontal axis. Figure 17A Plot the distance value D on the vertical axis, and... Figure 17B The actual object width W is plotted on the vertical axis. Figure 17B In this context, instead of the averaged actual object width W', the actual object width W calculated based on the object width at each time point is used. For ease of explanation, relative errors have been removed.

[0345] Even if Figure 17A As shown, when the relative distance value D changes, the actual object width W should also change accordingly. Figure 17B The dashed line in the figure indicates a constant distance. However, if absolute error is included, the actual object width W varies with distance, as shown by the solid line. The estimated distance value De with error can be represented by the true distance value Dt, the true actual object width Wt, and the estimated average actual object width We' using the following expression 12.

[0346]

[0347] If there is no relative error, then the relative distance can be correctly estimated because We and Wt are consistent and Dt and De are consistent. If We and Wt are inconsistent, i.e., an absolute error still exists, then an error also occurs in the relative distance calculated using the ratio of We to Wt. Therefore, as... Figure 17B As shown, the actual object width We changes based on the distance estimated.

[0348] Therefore, if a certain change occurs in the relative distance, an absolute distance correction component is determined in step S11608 to ensure that the actual object width remains unchanged at each moment. Since the true distance value Dt is unknown, the true distance value Dt can be estimated based on its variation component and the change in the actual object width W, taking into account any one or more of the grounding position ranging value D1, the camera surface phase difference ranging value D2, and the relative ranging value D.

[0349] Specifically, for example, the actual object width W is calculated using the camera phase difference ranging value D2 through a technique equivalent to that in step S1404, and its variation over time is calculated. The absolute error correction value can be corrected to reduce the variation in the actual object width W. Regarding the calculation method, general optimization techniques can be used to find the correction value that minimizes the variation in the actual object width W. Therefore, both the absolute error correction value and the actual object width correction value are estimated simultaneously.

[0350] One of the specific calculation methods will be explained. If it can be assumed that the error is included and the magnitude of the distance value is sufficiently large relative to the magnitude of the defocus, then the imaging expression, which is the relationship between the distance value and the defocus in the phase difference ranging of the image plane, and the relationship between the distance value and the image plane in the object width ranging value, are rearranged by the following expression 13.

[0351]

[0352] Since the actual vehicle width is ideally consistent regardless of distance, it is confirmed that, although not dependent on the distance value, the reciprocal of the actual vehicle width is proportional to the distance value by including the error. Regarding the relationship between 1 / Wt and D, the error in the actual vehicle width increases proportionally with distance, and the error decreases if the distance is small.

[0353] In other words, if the relative distance value D is approximately zero, then the error in Wt is approximately zero. For this reason, plotting the relative distance value D and the reciprocal of the actual vehicle width 1 / Wt yields an intercept that becomes the positive solution for the actual vehicle width, from which the absolute error correction value can be calculated.

[0354] In fact, since relative error is included in addition to absolute error, the plotting of data will vary. However, if the relative distance changes and there is data corresponding to a certain time, the least squares method can be used to solve it easily.

[0355] Furthermore, the absolute errors in the corrected camera phase difference ranging value D2, object width ranging value D3, and ground position ranging value D1 should be consistent with each other, and the difference between the absolute errors can also be calculated as the absolute error correction amount. The main reason for the absolute error component in the ground position ranging process in step S1401 includes the pitch angle deviation of the shooting device mentioned above. That is, the vanishing point is at a position different from the estimated position, and its component becomes the absolute error in the ground position ranging value.

[0356] In step S1608, since the absolute error component can be estimated, the pitch angle deviation can also be estimated. A correction amount can be used as the absolute error correction amount (data D1610). If the relative distance changes in this way, this information can be used to correct the absolute error component in each distance measurement with higher accuracy.

[0357] If the variation of the absolute error component, including the time series, is within a negligible range, then after the overall absolute error correction process in step S1608, the process can transition to the actual object width acquisition process in step S1504. In other words, essentially, the calculation of the average actual object width and the absolute error correction process in step S1607 are no longer needed. For this reason, the individual processes within this process can be simplified, or each process can be processed separately for continuous verification.

[0358] In step S1609, the distance information generation unit 322 selects which absolute error correction value from steps S1607 and S1608 to output as data D1608. As described above, if a certain change occurs in the relative distance, step S1608 is performed.

[0359] For this reason, in step S1609, the distance information generation unit 322 basically selects the absolute error correction value calculated in step S1607, and if the processing in S1608 is performed, it selects the absolute error correction value calculated in step S1608.

[0360] Again, if there are changes in the absolute error, the absolute error correction value calculated in step S1607 can be selected again. For example, if the absolute error correction value calculated in step S1607 changes after selecting the absolute error correction value calculated in step S1608, the absolute error correction value calculated in step S1607 can be selected again.

[0361] Subsequently, if the relative distance changes again, a better absolute error correction value can be continuously selected in the form of step S1608 and selecting its absolute error correction value.

[0362] As mentioned above, using Figures 15 to 17BThis paper describes the process of correcting relative and absolute errors over time using ground position ranging, object width ranging, and camera plane phase difference ranging. While the above process illustrates camera plane phase difference ranging, stereo ranging can also be considered similarly.

[0363] The relative error is equivalent to the effect of sensor noise, and the absolute error can be considered as the effect of the installation position errors of each stereoscopic imaging device. Although the absolute error is not converted into a defocus value, the correction amount can be estimated as the installation position error.

[0364] Alternatively, other forms such as LiDAR can be considered similarly. For example, in the case of LiDAR, since relative errors and absolute errors as range offsets occur similarly in range resolution, a concept similar to that of this embodiment can be applied.

[0365] Furthermore, while this presentation described the integrated flow of three types of ranging techniques, it is possible to extract the integrated flow of two types for each. Additionally, by adding camera-plane phase difference ranging, stereo, and LiDAR (other modalities), similar calibration can be achieved through a combination of four or more types of ranging techniques.

[0366] Because of this integrated ranging technology, stable ranging values ​​can be obtained for purposes such as tracking identified objects over a certain period of time. When the camera device is attached to a vehicle, it can be applied, for example, to adaptive cruise control (ACC) and autonomous driving.

[0367] In this way, in this embodiment, in addition to information from the common modality, image information can also be used to obtain object distance information of the corresponding object that has been identified with higher accuracy. Furthermore, in this embodiment, the path generation device 150 generates path information based on the integrated distance information generated in this manner.

[0368] Therefore, path information with higher accuracy can be generated. Here, the path generation device 150 is used as a path generation unit to perform the path generation step of generating path information based on integrated distance information. Regarding the path information, the path generation device 150 generates the path information based on the speed of the vehicle that is the mobile device.

[0369] <Third Embodiment>

[0370] The path generation apparatus 150 of the third embodiment achieves highly accurate ranging from short distances to long distances by combining parallax-based ranging of multiple images with distance estimation based on a single image.

[0371] Figure 18This is a block diagram illustrating an example of the construction of a ranging system according to a third embodiment. In this embodiment, the ranging system is included in a camera device 110. The ranging system includes an imaging element 1802 serving as an image acquisition unit, an identification processing unit 1821, a distance image generation unit 1812, a zoom ranging unit 1803, and a distance correction unit 1804.

[0372] Here, the imaging element 1802 has a similar structure to the imaging element 302 in the first embodiment. The identification processing unit 1821 corresponds to the identification processing unit 321. The distance image generation unit 1812 corresponds to the distance image generation unit 312. The zoom ranging unit 1803 and the distance correction unit 1804 correspond to the distance information generation unit 322.

[0373] The imaging element 1802 acquires image signals in a manner similar to that of the imaging element 302. The imaging element 1802 has a first photoelectric conversion unit 411 and a second photoelectric conversion unit 412 arranged within each pixel. In addition, it acquires an image signal composed of the image signal acquired by the first photoelectric conversion unit 411 and an image signal composed of the image signal acquired by the second photoelectric conversion unit 412.

[0374] These are images corresponding to different viewpoints and are called parallax images. Additionally, the imaging element 1802 acquires a composite image signal obtained by combining the image signals of the two parallax images as a captured image. The imaging element 1802 can acquire one of the two parallax images as a captured image.

[0375] Regarding the camera configuration for obtaining parallax images, stereo cameras arranged side-by-side can also be used. Furthermore, in a single-lens camera configuration, considering the speed of the host vehicle, parallax images can be obtained by assuming the relative movement of objects in consecutive frame images as parallax.

[0376] The recognition processing unit 1821 detects objects included in the captured image by applying image recognition processing to the captured image captured by the imaging element 1802. In order to realize the automatic driving control and collision reduction braking control of the vehicle 100, it is necessary to identify the traffic lane in which the vehicle 100 is traveling, the vehicle traveling in front of the vehicle 100 (the front vehicle), and objects such as people.

[0377] Regarding object detection technology, there are techniques that use template matching to detect objects with a nearly constant appearance (such as signals or traffic signs), or techniques that utilize machine learning to detect general objects (such as vehicles or people).

[0378] In this embodiment, the identification processing unit 1821 performs the segmentation line detection task and the object identification task. Figure 19 This is a schematic diagram illustrating an example of the output results of a segmentation line detection task and an object recognition task performed by the recognition processing unit 1821 according to the third embodiment. In the segmentation line detection task, when an image is input, a segmentation line region map with annotations is obtained (in this map, detectable segmentation lines are represented by black dashed lines). The annotations are used to indicate whether a pixel is a segmentation line (or a white line or a yellow line) on the road.

[0379] In the object recognition task, when an image is input, a machine learning model for detecting objects on the road is used to obtain the type of the detected object (person, car, or sign, etc.), the coordinates (x0, y0) of the top-left point in the detection box, and the coordinates (x1, y1) of the bottom-right point in the detection box. Furthermore, the coordinates of the detection boxes that are in contact with the detected object are obtained. Here, the output of the object recognition task is equivalent to the output of the external information described in the first embodiment.

[0380] The distance image generation unit 1812 obtains distance data from the parallax image obtained by the imaging element 1802. In the distance measurement based on the parallax image, the parallax value is calculated by detecting corresponding points between images with different viewpoints, and the distance can be calculated based on the parallax value and the camera conditions (focal length and baseline length) used to capture the parallax image.

[0381] As mentioned above, even if the camera that captured the parallax image is a single-lens camera using a dual-pixel CMOS sensor, the camera conditions used for distance calculation can still be identified. It is generally known that in distance measurement using parallax images, the accuracy of distance estimation deteriorates because the parallax almost disappears as the target moves further away.

[0382] The scaling distance measuring unit 1803 calculates the distance value of the second region by scaling the distance value of the first region calculated by the distance image generation unit 1812 based on the size comparison of objects in the first region and objects in the second region.

[0383] In this embodiment, an example of scaling the distance value on the short-distance side calculated by the distance image generation unit 1812 will be illustrated by extending the road surface on the short-distance side to the long-distance side based on the road information on the short-distance side and the long-distance side.

[0384] Figure 20This is an explanatory diagram illustrating an example of the positional relationship between the camera attachment position and the road surface. The road surface in the vertical direction of the camera attachment position V(0,Y0,0) is set as the origin O with global coordinates (X,Y,Z). Furthermore, in the imaging optical system 301 of the camera used in the imaging element 1802 and the optical axis of the imaging element, the optical axis direction is arranged horizontally.

[0385] Furthermore, due to the field of view of the camera, the area to be captured is represented as the capture range. When the capture range (that is, the image width of the captured image and the distance map calculated by the distance image generation unit 1812) is wu and the image height is hv, the center pixel of the distance map is represented as c(wu / 2, hv / 2).

[0386] First, the road surface on the short-distance side is estimated. The region in the lower part of the distance map is assumed to be the road surface on the short-distance side of the distance map. Alternatively, the road surface on the short-distance side can be directly estimated through recognition processing of the captured image. In the region in the lower part of the distance map, the area around the pixels identified as segmentation lines on the segmentation line region map detected by the recognition processing unit 1821 can be considered as the road surface.

[0387] Only distance map areas judged to be closer than the following thresholds can be considered as road surfaces. These thresholds are set based on information related to camera installation conditions (viewing angle, resolution, and line of sight), shooting environment (climate and time zone), and the road conditions on which the vehicle travels (lane width, number of lanes, branch points / intersections, and road type).

[0388] When the distance value of pixel p(u0,v0) on the distance map is D, the road surface p(u0,v0) on the short distance side can be transformed into X=u0-wu / 2, Y=v0-hv / 2 and Z=D using global coordinates.

[0389] When the road surface is assumed to be level, its equation is expressed as aX + bY + cZ + d = 0 (where a, b, c, and d are constants). Using four or more points representing the road surface on the shorter side, the road surface can be estimated by setting the constants in the aforementioned equation.

[0390] The road surface on the long-distance side can be estimated using the equation of the estimated road surface. Furthermore, this distance can be scaled based on the extended road surface on the long-distance side. Specifically, the depth Z of the point R(X,Y,Z) located at the intersection of the road surface equation and the line passing through the viewpoint V(0,Y0,0) and the pixel q(u1,v1) representing the region on the long-distance side of the distance map can be estimated as the distance value.

[0391] Even if the distance to the road corresponding to pixel q(u1,v1) cannot be obtained from the distance data, the scaling ranging unit 1803 can still obtain the distance value Z of the road corresponding to q(u1,v1) by calculating as described above.

[0392] Considering this processing of the distance map, the distance to the second region is calculated by scaling the distance of the first region based on the ratio (depth ratio in three-dimensional space) between the road surface present in the short-distance region (first region) and the road surface present in the long-distance region. Since the distance value calculated in this way does not use the distance values ​​of the parallax ranging units that deteriorate at a distance, the accuracy of the estimated distance at a distance is improved.

[0393] In addition, the detection frame detected by the identification processing unit 1821 can be used to improve the ranging accuracy. Figure 21 This is a schematic diagram illustrating an example of a scenario where two signs of known size are detected on a road. It is assumed that signs of pre-known size are placed on both the short-range and long-range sides within the acquired image.

[0394] In this state, the distance to the marker on the short-distance side can be accurately calculated using the distance image generation unit 1812. Furthermore, when the object size is known, the scaling distance measurement unit 1803 can calculate the distance to the marker on the long-distance side based on the ratio of the number of pixels in the image.

[0395] The height (number of pixels) of the marker on the short-range side is considered w0, and its distance relative to the camera position is considered d0. The height (number of pixels) of the marker on the long-range side is considered w1, and its distance relative to the camera position is considered d1. The zoom ranging unit 1803 can calculate the distance using d1 = d0 * (w0 / w1).

[0396] As mentioned above, if objects of known size (such as signs or traffic signals) can be detected, and objects of the same size on both the short-distance and long-distance sides (such as guardrails; or the width, length, and spacing of dividing lines; etc.) can be assumed to be the same size, then highly accurate zoom distance measurement can be performed.

[0397] Scaling distance measurement for objects of known size makes it difficult to improve accuracy across the entire distance range from near to far. However, accurate scaling distance measurement can be achieved by combining the aforementioned scaling distance measurement with the scaling distance measurement that utilizes the object size ratio.

[0398] The distance correction unit 1804 corrects the distance value measured by the distance image generation unit 1812 based on the distance value calculated by the scaling distance measurement unit 1803, and obtains the corrected distance value Dc. Hereinafter, the distance value measured by the distance image generation unit 1812 will be referred to as the distance value D, and the distance value calculated by the scaling distance measurement unit 1803 will be referred to as the scaling distance measurement value Ds. The corrected distance value Dc is calculated using the coefficient α according to the following expression 14.

[0399] Dc = α × D + (1 - α) × Ds....(Expression 14)

[0400] The coefficient α can be determined using any of the following three methods.

[0401] (1) A method for determining the coefficient α based on the magnitude of the distance value

[0402] Regarding the distance measurement of the parallax ranging unit, the distance value relative to the target itself affects the ranging accuracy. As the distance relative to the target increases, the accuracy of the distance value calculated by the parallax ranging unit decreases. Therefore, a determination coefficient α is used to increase the scaling ratio of the ranging value according to the distance value. Thus, accurate ranging results can be obtained. In other words, a determination coefficient α is used to decrease α as the distance value D increases.

[0403] (2) A method for determining the coefficient α based on the contrast of the ranging target.

[0404] Regarding the ranging of a disparity ranging unit, the contrast of the ranging target is a factor affecting ranging accuracy in addition to the distance value relative to the ranging target. When calculating corresponding points (disparity) between disparity images, even if matching is performed on regions with low contrast, the distinction between them and surrounding regions is unclear, thus the disparity cannot be accurately obtained.

[0405] For this reason, if the target is poorly illuminated at night and has low contrast, the distance measurement accuracy is judged to be low, and a coefficient α is set to increase the scaling ratio of the distance measurement value.

[0406] In other words, if the contrast of the ranging target is low, the determination coefficient α is increased. For example, since the scaled ranging value is generated based on the height-accurate distance value within the range illuminated by the vehicle's lighting, accurate ranging results can be obtained even if the contrast of the ranging target is low.

[0407] (3) A method for determining coefficients based on the type of the detection box.

[0408] Distance accuracy often varies between parallax ranging and zoom ranging depending on the type (category) of the target. For example, a target far from the ranging reference (such as the luminous part of a traffic light) can be ranged in parallax ranging without needing to be distinguished from other objects, but the accuracy often decreases in zoom ranging.

[0409] For this reason, if the detection box belongs to a specific type, accurate ranging results can be obtained by setting a coefficient to increase the proportion of the distance value. Additionally, as... Figure 21 As shown, if the scaling distance method increases the accuracy of the scaling distance measurement in the area surrounding the detected object, a coefficient can be determined so that the scaling distance measurement ratio around the detection box increases according to the category.

[0410] In the method described above for determining the coefficients, it is not necessary to limit the coefficients to 1, and the final coefficients can be determined based on the coefficients generated for each factor. Therefore, accurate distance measurement results can be obtained.

[0411] (A variation of the scaling distance measuring unit)

[0412] Regarding a variation of the scaling ranging unit 1803, an example will be given of estimating the distance to a target object by scaling adjacent distance data based on the ratio of the road widths on the short and long sides of the image. Figure 22 This is a block diagram showing the construction of a modified example of the scaling distance measuring unit 1803.

[0413] The zoom ranging unit 1803 includes a traffic lane analysis unit 2201, a roll angle estimation unit 2202, a ground contact position estimation unit 2203, and an object distance calculation unit 2204. The traffic lane analysis unit 2201 detects the number of pixels between the dividing lines from the dividing line area map as the width of the traffic lane, and detects the center coordinates between the dividing lines as the center (coordinates) of the traffic lane.

[0414] Figures 23A to 23D This is a schematic diagram illustrating an example of the processing of lane width detection and lane center detection performed by the lane analysis unit 2201 according to the third embodiment. Figure 23A This is a schematic illustration of a region representing a segmentation line in an image.

[0415] The traffic lane analysis unit 2201 assigns segmentation line markers (represented by black dashed lines in the image) to pixels detected as segmentation lines in the image. Regarding segmentation lines on the road, since the boundary between the driving lane and the overtaking lane can be represented by dashed lines, or the segmentation lines may be hidden due to cars or obstacles on the road, the segmentation line region map appears intermittently.

[0416] Figure 23B This is a schematic diagram illustrating the technique for calculating the width and center position of a traffic lane. The traffic lane analysis unit 2201 sequentially checks from left to right in the segmentation area diagram whether a segmentation mark has been set for each pixel. The starting point for the road width is the pixel being checked if it does not have a segmentation mark, and if the pixel to its left adjacent pixel does have a segmentation mark.

[0417] Additionally, the endpoint of the road width is a pixel in the following check: this pixel has a dividing line marker, and the pixel to its left adjacent pixel does not have a dividing line marker. Therefore, as... Figure 23B As shown by the arrow group, a lane width (arrow length) and a lane location (arrow center position) can be detected for a line in the dividing line area map.

[0418] Figure 23C This is a schematic diagram illustrating the detected width data of the open traffic lane. As long as the actual size of the open traffic lane remains unchanged, the lane width is observed using the number of pixels proportional to the inverse of the distance relative to the camera. Figure 23C In the diagram, the horizontal axis represents the lines of the segmentation area map, and the vertical axis represents the detected width of the open lane (in pixels).

[0419] In this way, the width of the open lane has a highly linear relationship with respect to the line. Although the width of one open lane and the width of two open lanes can be observed in a mixed manner based on the intermittent state of the dividing line, the two are easily separated because there is a difference of about twice between them.

[0420] Figure 23D This illustration shows the interpolation of observation data for a separated left-hand lane and data from lines where the lane width cannot be obtained from the observation data. For the separation of adjacent lanes (left and right lanes), observation data for adjacent lanes can be easily excluded by comparing the positions of the lanes (center of the arrow). Based on this information, robust estimation can be performed, for example, using methods such as RANSAC. Figure 23B The width of the single-lane roadway shown on the left.

[0421] For interpolation of the width of open lanes, an approximate straight-line equation can be calculated using methods such as RANSAC described above. An approximate straight-line expression can be used, or interpolation can be performed to interpolate the road width data between the observed two points. Similarly, the location of open lanes can be obtained using methods similar to those described above. The obtained open lane information (open lane width and open lane center) is then used for scaling distance measurement.

[0422] The roll angle estimation unit 2202 estimates the roll angle of the camera based on the distance map. Several factors influence the roll angle generated by the vehicle-mounted camera. For example, uneven road surfaces may cause a height difference between the contact surfaces of the left and right tires, which could lead to a roll angle due to the inability to maintain a horizontal attachment position of the camera.

[0423] In addition, the vehicle itself may deform due to the centrifugal force when turning around a curve, potentially causing a roll angle. Such a roll angle significantly affects distance estimation.

[0424] The scaling distance measurement described in this article is processed as follows: For the distance to an object on the road surface, the accurately measured adjacent distance value is scaled using the ratio of the number of pixels in the lane assumed to be at the same distance as the object to the number of pixels in the adjacent lane. If this causes a camera tilt angle, it becomes difficult to find the lane width assumed to be at the same distance as the object.

[0425] Figure 24 This is a schematic diagram comparing the presence or absence of a roll angle between positions of the lane width in captured images. The image on the left illustrates the case where no roll angle is generated, and the image on the right illustrates the case where a 10° roll angle is generated centered on the lower left side of the image in the same scene. In the case of the image on the left, since the ground contact point of the car (range-finding target) and the lane, which is assumed to exist at the same distance, are arranged in a straight line, countermeasures can be easily taken.

[0426] On the other hand, in the case of an image taken on the right side that produced the roll angle, there is no lane for traffic on the same line as the car (the ranging target)'s contact point, and even if a lane for traffic exists, the lane width at the same distance cannot be calculated. To determine the lane width at the same distance as the car (the ranging target), the roll angle needs to be accurately estimated.

[0427] The roll angle estimation unit 2202 estimates the roll angle based on a distance map obtained through parallax ranging. If a roll angle is generated, the distance to a road surface that can be assumed to be level is estimated, such as... Figure 24 As shown, the positions at the same distance are tilted.

[0428] Because the tilt angle pivots around the viewpoint, the distance between the camera and the target object remains unchanged, and the image captured by the camera is tilted due to the tilt angle. Therefore, the distance map is also tilted as if it were tilted along the road surface in the image.

[0429] Figure 25 This is a flowchart illustrating an example of the roll angle estimation process performed by the roll angle estimation unit 2202 according to the third embodiment. Figures 26A to 26C These are illustrative schematic diagrams illustrating various processing examples in the roll angle estimation process according to the third embodiment.

[0430] By causing the computer program stored in the computer execution memory included in the path generation device 150 to perform... Figure 25 The flowchart shows the steps involved.

[0431] In step S2500, the tilt angle estimation unit 2202 determines the observation pixel setting range based on the distance map. Figure 26A This is a schematic diagram illustrating the range of observation pixels in a distance graph.

[0432] like Figure 26A As shown, the observation pixel setting range is positioned in the lower left region of the distance map. The height of the observation pixel setting range is assumed to be in the region corresponding to the road surface, and for example, is set from the horizontal line to the lower end of the distance map. The width of the observation pixel setting range is set taking into account the interval between the observation pixels and the search range. The search range is set at a position that is horizontally away from the observation pixels by a predetermined number of pixels (a predetermined interval).

[0433] If the interval between the observed pixel and the search area is large, the search area cannot be set even if a large width is set for the observed pixel setting range. In this embodiment, the predetermined interval is set to approximately 1 / 4 of the image width of the distance map, and the size of the observed pixel setting range is set to approximately half of the image width of the distance map. The computational load can be controlled by appropriately setting the observed pixel setting range.

[0434] In step S2501, the tilt angle estimation unit 2202 obtains distance data of the observed pixel 2600 from the distance map. In this embodiment, the distance data is obtained relative to the observed pixel 2600 within the set range of the observed pixel.

[0435] In step S2502, the tilt angle estimation unit 2202 determines the search range. Regardless of whether a tilt angle is generated, the distance values ​​of pixels near the observed pixel are often closer to the distance value of the observed pixel. Therefore, in order to detect a high-resolution tilt angle based on the coordinates of the observed pixel 2600 and the corresponding pixel 2601, the tilt angle estimation unit 2202 sets the search range as a region with a predetermined interval in the horizontal direction relative to the observed pixel 2600.

[0436] The roll angle estimation unit 2202 can limit the height of the search range by estimating the range of the generated roll angle. For example, in the case of a vehicle-mounted camera installed in a vehicle, if the vehicle is traveling on a regular road, the roll angle is limited to ± a few degrees. In this embodiment, approximately 1 / 8 of the image height in the distance map is set.

[0437] The search range is set from a position horizontally away from the observed pixel 2600 by a predetermined interval to the right end of the distance map. The right end of the search range is not limited to the right end of the distance map. However, if the search range is small, the area becomes too small, making it impossible to find the road distance data corresponding to the observed pixel 2600. Therefore, it is preferable to set the search range as large as possible.

[0438] In step S2503, the tilt angle estimation unit 2202 searches for a corresponding pixel 2601 from the search range that corresponds to the observed pixel 2600. To position a pixel with a distance value similar to that of the observed pixel 2600 within the search range as the corresponding pixel 2601, the difference between the distance values ​​of the corresponding pixel 2601 and the observed pixel 2600 is detected for each pixel within the search range, and the pixel with the smallest difference is selected as the corresponding pixel 2601.

[0439] The method for searching for the corresponding pixel 2601 is not limited to a difference comparison between pixels, and the central pixel in the pixel group with the highest similarity can be selected as the corresponding pixel 2601 by performing a difference comparison between the distance values ​​of adjacent pixel groups including the observed pixel and the pixels in the search range.

[0440] In step S2504, the tilt angle estimation unit 2202 calculates the tilt angle θ based on the coordinates of the observed pixel and the corresponding pixel. For example... Figure 26B As shown, when the coordinates of the observed pixel 3800 are (x0, y0) and the coordinates of the corresponding pixel 2601 are (x1, y1), the tilt angle θ is calculated by θ = arctan((y1-y0) / (x1-x0)).

[0441] In step S2505, the tilt angle estimation unit 2202 branches based on whether all processing within the observation pixel setting range has been completed. If processing is completed with all pixels within the observation pixel setting range as observation pixels, the process proceeds to step S2506. If processing is not completed with all pixels within the observation pixel setting range as observation pixels, the process proceeds to step S2501, and processing is performed for the new observation pixel.

[0442] In step S2506, the roll angle estimation unit 2202 calculates the roll angle. This process is repeated periodically. Figure 25 The process continues until the user issues an end command (not shown).

[0443] Figure 26C This is a schematic diagram illustrating the tilt angle calculated when the horizontal axis represents the distance value of the observed pixel and the vertical axis represents the observed pixel. Since the tilt angle calculated based on a single observed pixel includes a noise component, the likelihood tilt angle is detected by averaging multiple tilt angles.

[0444] Although a distance map is generated from the disparity image for tilt angle estimation, it is known that the detection accuracy of ranging based on the disparity image deteriorates as the distance to the object increases.

[0445] For this reason, when calculating the tilt of each observed pixel using a distance map, the non-uniformity of the tilt calculated based on the magnitude of the distance value of the observed pixel increases. For this reason, in the calculation of the tilt angle, a weighted average is used to estimate the tilt angle based on the increasing proportion of observed pixels with small tilt values ​​and the decreasing proportion of observed pixels with large tilt values.

[0446] The factor used to determine the weighted average ratio can be the similarity when searching for corresponding pixels. Through the above processing, the roll angle can be estimated using a distance map of the road surface.

[0447] In addition, a method for setting the interval between the observation pixels and the search range with the estimated tilt angle resolution predetermined will be explained.

[0448] The interval between the observed pixel and the search range is set by the resolution of the tilt angle. The tilt angle is calculated based on the inclination from the observed pixel to the corresponding pixel and is represented by the ratio of the horizontal difference between the observed pixel and the corresponding pixel to the vertical difference between them. Since the vertical difference is at least one pixel, the resolution of the tilt angle is determined by the magnitude of the horizontal difference. This can be expressed as follows:

[0449] Equation 15 uses the resolution r of the tilt angle to obtain the interval d between the observed pixel and the search range.

[0450]

[0451] Mathematical expression 15 represents the minimum detection angle and is derived from the relationship between the angle r (considering a difference of one pixel in the y-axis direction) and the interval d between the observed pixel and the search range. A larger calculated interval d results in higher resolution, but as d increases, the search range narrows, as mentioned above. Therefore, it is preferable to set a minimum resolution.

[0452] For example, if the required resolution r is 0.1°, then the interval d becomes 573 pixels or more. If the resolution used for detecting the tilt angle is applied using the above calculation method, an appropriate interval can be set.

[0453] The grounding position estimation unit 2203 uses the detection box obtained by the object detection task, the lane center obtained by the lane analysis unit 2201, and the roll angle estimated by the roll angle estimation unit 2202 to estimate the coordinates of the lane width data located at the same distance as the grounding position of the ranging target.

[0454] Figure 27 This is a flowchart illustrating an example of coordinate estimation processing of lane width data performed by the grounding position estimation unit 2203 according to the third embodiment. This is performed by causing the computer included in the path generation apparatus 150 to execute a computer program stored in its memory. Figure 27 The flowchart shows the steps involved.

[0455] In step S2701, the grounding position estimation unit 2203 selects the detection box of the ranging target from the detection boxes obtained through the object identification task.

[0456] In step S2702, the grounding position estimation unit 2203 sets the center position of the lower part of the detection frame as the coordinates (xc, yc) of the grounding position of the ranging target.

[0457] Figure 28 This is a schematic diagram illustrating an example of the coordinates (xc, yc) of the grounding position of the ranging target set by the grounding position estimation unit 2203 according to the third embodiment. The coordinates (xc, yc) of the grounding position of the ranging target are represented by the upper left coordinates (x0, y0) and lower right coordinates (x1, y1) of the detection frame using the following expression 16.

[0458]

[0459] Regarding the method for obtaining the coordinates of the ground contact point of the ranging target, different techniques may also be used. In the case of objects with a large lateral width, such as cars, which are photographed with a tilted side angle, the deviation between the center coordinates at the lower end of the detection frame and the coordinates of the tire actually contacting the ground may increase.

[0460] In this case, for example, the following construction can be used: the outline of the object within the detection box is detected using an image or distance map, and the position where a straight line R with a tilt angle contacts the lower end of the object's outline is set as the ground contact coordinate. The coordinates of the actual contact between the ranging target and the ground can be set using such a construction.

[0461] Since processing the outline of the object to be identified requires a large computational load, in order to consider the computational cost of the construction, a suitable coordinate can be selected from the angular coordinates of the detection box based on the positional relationship between the center of the traffic lane and the ranging target.

[0462] Specifically, there is a selection method that selects the coordinates of the lower left side of the detection frame when the ranging target is located to the left of the center of the traffic lane, and selects the coordinates of the lower right side of the detection frame when the ranging target is located to the right. Therefore, the coordinates where the ranging target actually contacts the straight line R obtained in the next step can be set to coordinates closer to the coordinates set at the lower center of the detection frame than the coordinates set at the lower center of the detection frame.

[0463] In step S2703, the grounding position estimation unit 2203 obtains coordinates (xt, yt), which are obtained by passing the coordinates (xc, yc) of the grounding position of the ranging target, and in these coordinates, the straight line R whose inclination becomes the side tilt angle intersects with the straight line C of the center of the traffic lane obtained by the dividing line detection.

[0464] Through the processing described above, even if the distance map is tilted due to the tilt angle, the coordinates of the lane width data at the same distance as the grounding position of the ranging target can be obtained.

[0465] The object distance calculation unit 2204 uses coordinates related to the lane width data corresponding to the ground position obtained by the ground position estimation unit 2203, lane width data calculated by the lane analysis unit 2201, and data related to the distance map to calculate the distance to the ranging target.

[0466] Figure 29 This is a flowchart illustrating an example of the distance estimation process to the ranging target performed by the object distance calculation unit 2204 according to the third embodiment. This is performed by causing the computer included in the path generation apparatus 150 to execute a computer program stored in its memory. Figure 29 The flowchart shows the steps involved.

[0467] In step S2901, the object distance calculation unit 2204 uses the lane width data and coordinates (xt, yt) to calculate the lane width N2. The object distance calculation unit 2204 can obtain the lane width at coordinates (xt, yt) as the lane width N2, and can obtain the lane width based on a weighted average by obtaining multiple lane widths near coordinates (xt, yt) and weighting them according to their distances relative to coordinates (xt, yt).

[0468] If the lane width is obtained from the lanes detected by image recognition, there is a possibility that the lane width detected based on errors in image recognition will be blurred with the actual lane width. As mentioned above, the blurring of the detected lane width can be reduced by calculating the lane width N2 using multiple lane widths. The number of lane width data used for smoothing can be one, depending on the processing load.

[0469] In step S2902, the object distance calculation unit 2204 retains the lane width data and distance data obtained from the current frame in a mapping manner. First, the object distance calculation unit 2204 obtains the lane width data and its coordinates for each lane, and obtains the distance data corresponding to the coordinates of the lane width data from the distance map.

[0470] At this point, the corresponding distance data can be obtained by acquiring the distance data corresponding to each coordinate within the range of lane width data (xn-N1 / 2,yn) to (xn+N1 / 2,yn) centered on the coordinate (xn,yn) corresponding to the lane width N1 of the lane centerline C, and smoothing the distance data.

[0471] The object distance calculation unit 2204 performs this mapping between the lane width data and distance data of each lane width data obtained in frame T0 at the current time. Figure 30 This is a schematic diagram illustrating an example of lane width data according to the third embodiment. The object distance calculation unit 2204 can narrow down the range of the amount of data to be mapped to each other based on the processing load or recording capacity. For example, it can be configured to retain only the data at locations with a high probability of having distance data.

[0472] In step S2903, the object distance calculation unit 2204 calculates the reference data B used for calculating the distance to the ranging target. When the number of data mapped in the previous step is k, the reference data B is represented by the distance D1 to the traffic lane and the width N1 of the traffic lane using the following expression 17.

[0473]

[0474] At this point, the error in the distance data may increase as the distance increases. Therefore, the baseline data B can be calculated by a weighted average that increases the weight corresponding to parts with small distance values ​​and decreases the weight corresponding to parts with large distance values. Alternatively, the data obtained in step S2802 can be configured such that the data corresponding to several frames such as T1 and T2 are kept as past frame data, and the calculation can be performed with this data included.

[0475] In mathematical expression 16, D1[n] × N1[n] becomes a value corresponding to the width of the open lane as the reference for distance calculation. When the distance to the open lane is D, the number of pixels representing the width of the open lane in the image is N pixels, the horizontal field angle of the camera is F°, and the horizontal image width is H pixels, the actual width of the open lane W can be calculated using W = (D × N) × 2 × tan(F / 2) / W. When the camera specifications are determined, the portion 2 × tan(F / 2) / W can be treated as a fixed value.

[0476] Therefore, the actual lane width becomes a variable of (D×N). Due to this relationship, in mathematical expression 16, since the lane width can be considered the same when the vehicle is in motion, observation noise can be suppressed by smoothing over D1[n]×N1[n].

[0477] Furthermore, even in the case of multiple consecutive frames, since it is assumed that the vehicles are traveling in the same open lane, the actual open lane width can be regarded as the same, and thus observation noise can be suppressed by performing smoothing on D1[n]×N1[n].

[0478] On the other hand, there is a possibility that when the vehicle body rides on small stones, etc., it will become blurred in the image according to the frame, making it impossible to accurately obtain the lane width data. For this reason, calculations can be performed to reduce the error relative to the reference data by reducing the weight of frames with large deviations (that is, large blurs) in D1[n]×N1[n].

[0479] Alternatively, calculations can be performed to maintain a balance between the response to the frame relative to the current moment and the smoothing of fuzziness generated between frames by increasing the weight as the time difference between the frames decreases and decreasing the weight as the time difference increases. Furthermore, since the reference data also becomes roughly uniform during periods when the lane width does not change significantly, the system can be structured such that processing load is suppressed by utilizing previously acquired reference data and skipping the aforementioned calculations (mathematical expression 17) during such periods.

[0480] In step S2904, the object distance calculation unit 2204 uses the reference data B obtained in the previous step and the lane width N2 of the ranging target to obtain the distance data D2 of the ranging target. The distance data D2 can be obtained by the following expression.

[0481] D2 = B / N2....(Expression 18)

[0482] Through the processing described above, the distance to the ranging target can be estimated. The object distance calculation unit 2204 outputs object distance information representing the distance to the ranging target obtained in this manner.

[0483] As described above, this embodiment illustrates an example of a ranging system capable of measuring distances with high accuracy from near to far by combining stereo ranging using a parallax image with distance estimation from a single image. Even when the ranging unit based on the parallax image is located at a distance where ranging accuracy deteriorates, distance accuracy can be maintained even when the ranging target is far away by using the scaling ratio of objects such as the road surface.

[0484] If the target object to be scaled is an object such as a road whose actual size is unlikely to change, distance accuracy can be improved by reducing observation noise when measuring size through smoothing during scaling.

[0485] Furthermore, more robust distance estimation can be achieved by varying the synthesis ratio based on the distance range in both parallax-based ranging and single-image-based distance estimation. Additionally, robust distance estimation can be further improved by detecting and correcting roll angles, thus mitigating the effects of vehicle deformation or uneven road surfaces.

[0486] <Other Embodiments>

[0487] In the above embodiments, an imaging device for obtaining distance image data by acquiring parallax images of the left and right sides via the same optical system using the phase difference method of the imaging surface has been described as an example, but the acquisition method is not limited thereto.

[0488] The parallax images on the left and right sides can also be obtained by using a so-called stereo camera, which consists of two cameras positioned at a predetermined distance from each other on the left and right sides to acquire parallax images on the left and right sides respectively.

[0489] Alternatively, distance information can be acquired using ranging devices such as LiDAR, and the aforementioned ranging can also be performed using external information obtained through image recognition of images captured from a camera device. The above embodiments can be appropriately combined. Furthermore, in the above embodiments, integrated distance information can be generated based on the history of at least two of the object's first to third distance information, or the history of integrated distance information.

[0490] In the above embodiments, an example of assembling a distance calculation device as an electronic instrument in a vehicle that is a car (mobile device) has been described. However, the mobile device can be any mobile device, such as an autonomous two-wheeled vehicle, a bicycle, a wheelchair, a ship, an airplane, a drone, or a mobile robot such as an AGV or AMR, as long as it can move.

[0491] Furthermore, the distance calculation device used as the electronic instrument in this embodiment is not limited to distance calculation devices assembled in these mobile bodies, but includes distance calculation devices that acquire images from cameras or the like assembled in the mobile device via communication and calculate distances at locations far from the mobile device.

[0492] Although the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims is to be interpreted in the broadest sense to include all such modifications, equivalent structures, and functions.

[0493] Furthermore, as part or all of the control according to the embodiments, a computer program that implements the functions of the above embodiments can be supplied to an electronic instrument via a network or various storage media. The computer (or CPU or MPU, etc.) of the electronic instrument can then be configured to read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention.

[0494] This application claims the benefit of Japanese Patent Application 2022-036621, filed on March 9, 2022, which is incorporated herein by reference in its entirety.

Claims

1. An electronic instrument, comprising: At least one processor or circuit is configured to function as: The first distance information acquisition unit is configured to acquire first distance information corresponding to the object included in the image signal; The second distance information acquisition unit is configured to acquire second distance information based on the end position information of the object included in the image signal; The third distance information acquisition unit is configured to acquire third distance information based on information about the size of the objects included in the image signal. as well as The distance information integration unit is configured to generate integrated distance information by combining and integrating the first distance information, the second distance information, the third distance information, and the distance measurement value from the dynamic model based on the relative velocity parameter relative to the object.

2. The electronic instrument according to claim 1, in, The distance information integration unit generates the integrated distance information by performing a weighted sum of at least two of the first distance information, the second distance information, and the third distance information.

3. The electronic instrument according to claim 1, in, The distance information integration unit generates the integrated distance information by selecting at least one of the first distance information, the second distance information, and the third distance information.

4. The electronic instrument according to claim 1, in, The first distance information acquisition unit acquires the first distance information by using a phase difference ranging method.

5. The electronic instrument according to claim 4, in, The first distance information acquisition unit acquires the first distance information based on signals from the first photoelectric conversion unit and the second photoelectric conversion unit arranged within the pixel of the camera element, using the phase difference ranging method.

6. The electronic instrument according to claim 4, in, The first distance information acquisition unit acquires the first distance information based on two image signals from a stereo camera using the phase difference ranging method.

7. The electronic instrument according to claim 1, in, The second distance information acquisition unit acquires the second distance information based on the ground position or bottom position information of the object included in the image signal.

8. The electronic instrument according to claim 7, in, The second distance information acquisition unit has a tilt angle estimation unit, which is used to estimate the tilt angle of the camera used to acquire the image signal.

9. The electronic instrument according to claim 1, in, The third distance information acquisition unit acquires the third distance information based on the width or height information of the object included in the image signal.

10. The electronic instrument according to claim 1, in, The distance information integration unit generates the integrated distance information based on the type of the object.

11. The electronic instrument according to claim 1, in, The distance information integration unit generates the integrated distance information based on the history of at least two of the object's first distance information, second distance information, and third distance information.

12. The electronic instrument according to claim 1, in, The at least one processor or circuit is also configured to function as: A fourth distance information acquisition unit is configured to use radar to acquire fourth distance information about the object, and The distance information integration unit generates the integrated distance information based on the fourth distance information.

13. A mobile device, in, At least one processor or circuit is also configured to function as: A path generation unit is configured to generate path information based on the integrated distance information acquired by the electronic instrument according to claim 1.

14. The mobile device according to claim 13, in, At least one processor or circuit is also configured to serve as a path generation unit configured to generate path information based on the integrated distance information and the speed of the mobile device.

15. The mobile device according to claim 13, in, At least one processor or circuit is also configured to function as: A drive control unit is configured to control the drive of the mobile device based on path information generated by the path generation unit.

16. A distance calculation method, comprising: The first distance information acquisition step is used to acquire first distance information corresponding to the objects included in the image signal; The second distance information acquisition step is used to acquire second distance information based on the end position information of the object included in the image signal; The third distance information acquisition step is used to acquire third distance information based on the size information of the objects included in the image signal; as well as The distance information integration step is used to generate integrated distance information by combining and integrating the first distance information, the second distance information, the third distance information, and the distance measurement value from the dynamic model based on the relative velocity parameter relative to the object.

17. A non-transitory computer-readable storage medium configured to store a computer program including instructions for performing the following processes: The first distance information acquisition step is used to acquire first distance information corresponding to the objects included in the image signal; The second distance information acquisition step is used to acquire second distance information based on the end position information of the object included in the image signal; The third distance information acquisition step is used to acquire third distance information based on the size information of the objects included in the image signal; as well as The distance information integration step is used to generate integrated distance information by combining and integrating the first distance information, the second distance information, the third distance information, and the distance measurement value from the dynamic model based on the relative velocity parameter relative to the object.

18. A computer program product comprising a computer program, the computer program including instructions for performing the following processes: The first distance information acquisition step is used to acquire first distance information corresponding to the objects included in the image signal; The second distance information acquisition step is used to acquire second distance information based on the end position information of the object included in the image signal; The third distance information acquisition step is used to acquire third distance information based on the size information of the objects included in the image signal; as well as The distance information integration step is used to generate integrated distance information by combining and integrating the first distance information, the second distance information, the third distance information, and the distance measurement value from the dynamic model based on the relative velocity parameter relative to the object.

Images