Object Detection Device, Object Detection System, Moving Body, and Object Detection Method

Abstract

The processor of the object detection device is configured to search for coordinates corresponding to the object disparity along the second direction of the disparity map and update them to the first coordinates, and calculate the height of the object corresponding to the object disparity based on the first coordinates. The disparity map is a map in which two-dimensional coordinates composed of the first direction and the second direction are associated with disparities. The processor is configured to calculate a first candidate height based on the first coordinates and a second candidate height based on the candidate coordinates when there are candidate coordinates corresponding to a disparity approximately equal to the object disparity at a distance exceeding a specified interval from the first coordinates. The processor is configured to determine which of the first candidate height and the second candidate height is to be taken as the height of the object based on the disparities respectively corresponding to two coordinates sandwiching the candidate coordinates in the first direction of the disparity map.

Description

[0001] Cross - reference to related applications

[0002] This application claims the priority of Japanese Patent Application No. 2019 - 170902 filed in Japan on September 19, 2019, and incorporates all the disclosures of the prior application herein by reference. Technical field

[0003] The present invention relates to an object detection device, an object detection system, a moving body, and an object detection method. Background art

[0004] In recent years, an object detection device using a stereo camera has been mounted on a moving body such as an automobile. Such an object detection device acquires a plurality of images from the stereo camera and detects an object that may be an obstacle based on the acquired plurality of images (for example, refer to Patent Document 1).

[0005] Prior art documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Unexamined Patent Publication No. 5 - 265547 Summary of the invention

[0008] An object detection device according to an embodiment of the present invention includes a processor. The processor is configured to search for coordinates corresponding to an object parallax that satisfies a specified condition along a second direction of a parallax map and update them to first coordinates, and calculate the height of an object corresponding to the object parallax based on the first coordinates. The parallax map is a map in which a two - dimensional coordinate composed of a first direction corresponding to the horizontal direction of a captured image generated by a stereo camera capturing a road surface and a second direction intersecting the first direction is associated with a parallax obtained from the captured image. The processor is configured to, when there are candidate coordinates corresponding to a parallax approximately equal to the object parallax at a position exceeding a specified interval from the first coordinates, calculate a first candidate height based on the first coordinates and calculate a second candidate height based on the candidate coordinates. The processor is configured to determine which of the first candidate height and the second candidate height is taken as the height of the object based on the parallaxes respectively corresponding to two coordinates sandwiching the candidate coordinates in the first direction of the parallax map.

[0009] An object detection system according to an embodiment of the present invention includes: a stereo camera that captures a plurality of images having parallax with each other; and an object detection device including at least one processor. The processor is configured to search for coordinates corresponding to an object parallax that satisfies a specified condition along a second direction of a parallax map and update them to first coordinates, and calculate the height of an object corresponding to the object parallax based on the first coordinates. The parallax map is a map in which a two-dimensional coordinate system composed of a first direction corresponding to the horizontal direction of a captured image generated by the stereo camera capturing a road surface and a second direction intersecting the first direction is associated with parallax obtained from the captured image. The processor is configured to, when there are candidate coordinates corresponding to a parallax that is approximately equal to the object parallax at a position exceeding a specified interval from the first coordinates, calculate a first candidate height based on the first coordinates and calculate a second candidate height based on the candidate coordinates. The processor is configured to determine which of the first candidate height and the second candidate height is to be obtained as the height of the object based on the parallax respectively corresponding to two coordinates sandwiching the candidate coordinates in the first direction of the parallax map.

[0010] A moving body according to an embodiment of the present invention has an object detection system that includes: a stereo camera that captures a plurality of images having parallax with each other; and an object detection device including at least one processor. The processor is configured to search for coordinates corresponding to an object parallax that satisfies a specified condition along a second direction of a parallax map and update them to first coordinates, and calculate the height of an object corresponding to the object parallax based on the first coordinates. The parallax map is a map in which a two-dimensional coordinate system composed of a first direction corresponding to the horizontal direction of a captured image generated by the stereo camera capturing a road surface and a second direction intersecting the first direction is associated with parallax obtained from the captured image. The processor is configured to, when there are candidate coordinates corresponding to a parallax that is approximately equal to the object parallax at a position exceeding a specified interval from the first coordinates, calculate a first candidate height based on the first coordinates and calculate a second candidate height based on the candidate coordinates. The processor is configured to determine which of the first candidate height and the second candidate height is to be obtained as the height of the object based on the parallax respectively corresponding to two coordinates sandwiching the candidate coordinates in the first direction of the parallax map.

[0011] An object detection method according to an embodiment of the present invention includes the following steps: Search along the second direction of the disparity map for coordinates corresponding to an object disparity that satisfies a specified condition and update them to first coordinates, and calculate the height of an object corresponding to the object disparity based on the first coordinates. The disparity map is a map in which a two-dimensional coordinate system composed of a first direction corresponding to the horizontal direction of a captured image generated by a stereo camera capturing a road surface and a second direction intersecting the first direction is made to correspond to disparities obtained from the captured image. The step of calculating the height of the object includes: when there are candidate coordinates corresponding to a disparity approximately equal to the object disparity at a specified interval from the first coordinates, calculate a first candidate height based on the first coordinates and calculate a second candidate height based on the candidate coordinates. The step of calculating the height of the object includes: based on disparities respectively corresponding to two coordinates sandwiching the candidate coordinates in the first direction of the disparity map, determine which of the first candidate height and the second candidate height is to be taken as the height of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram showing a schematic configuration of an object detection system according to an embodiment of the present invention.

[0013] Figure 2 is a schematic side view of a moving body equipped with Figure 1 the object detection system shown.

[0014] Figure 3 is a schematic front view of a moving body equipped with Figure 1 the object detection system shown.

[0015] Figure 4 is a block diagram showing a schematic configuration of an object detection system according to another embodiment of the present invention.

[0016] Figure 5 represents Figure 1 an example of a flowchart of a process executed by the object detection device shown.

[0017] Figure 6 is a diagram illustrating an example of a first disparity image acquired or generated by the object detection device.

[0018] Figure 7 is a flowchart showing an example of a process for estimating the shape of a road surface.

[0019] Figure 8 is a flowchart showing an example of a process for extracting a road surface candidate disparity from a first disparity image.

[0020] Figure 9It is a diagram showing the positional relationship between the road surface and the stereo camera.

[0021] Figure 10 It is a diagram explaining the steps for extracting the candidate disparity of the road surface.

[0022] Figure 11 It is a diagram showing the range on the road surface where the disparity has been histogrammed.

[0023] Figure 12 It is a d - v correlation diagram showing an example of the relationship between the road surface disparity d r and the vertical coordinate (v - coordinate).

[0024] Figure 13 It is a diagram explaining the method for detecting whether there are objects that are not road surface disparities.

[0025] Figure 14 It is a flowchart of the process for approximating the relationship between the road surface disparity d r and the vertical coordinate (v - coordinate) of the image using a straight line.

[0026] Figure 15 It is a diagram explaining the approximation of the road surface disparity d using the first straight line r

[0027] Figure 16 It is a diagram explaining the method for determining the second straight line.

[0028] Figure 17 It is a diagram showing an example of the result of approximating the relationship between the road surface disparity d r and the vertical coordinate (v - coordinate) of the image using a straight line.

[0029] Figure 18 It is a diagram showing an example of the second disparity image.

[0030] Figure 19 It is a reference diagram of the second disparity image.

[0031] Figure 20 It is a flowchart showing an example of the detection process for the first disparity and the second disparity.

[0032] Figure 21 It is Figure 18 a diagram showing the overlapping display of a partial area on the second disparity image shown.

[0033] Figure 22 It is a diagram showing an example of the disparity histogram.

[0034] Figure 23 It is a flowchart showing an example of the calculation process for calculating the height of an object (Part 1).

[0035] ​Figure 24 It is a flowchart (part 2) showing an example of a calculation process for calculating the height of an object.

[0036] Figure 25 It is a flowchart (part 3) showing an example of a calculation process for calculating the height of an object.

[0037] Figure 26 It is a diagram showing an example of a second parallax image.

[0038] Figure 27 It is a diagram showing an example of a second parallax image.

[0039] Figure 28 It is related to Figure 27 The first image corresponding to the second parallax image shown.

[0040] Figure 29 It is a diagram showing an example of a second parallax image.

[0041] Figure 30 It is related to Figure 29 The first image corresponding to the second parallax image shown.

[0042] Figure 31 It is a flowchart showing an example of a detection process for parallel objects.

[0043] Figure 32 It is a diagram showing an example of a UD diagram.

[0044] Figure 33 It is related to Figure 32 The first image corresponding to the UD diagram shown.

[0045] Figure 34 It is a diagram of a UD diagram from which the point group substantially parallel to the u direction has been removed.

[0046] Figure 35 It is a diagram (part 1) explaining the Hough Transform.

[0047] Figure 36 It is a diagram (part 2) explaining the Hough Transform.

[0048] Figure 37 It is a flowchart showing an example of a restoration process.

[0049] Figure 38 It is a diagram showing an example of a UD diagram.

[0050] Figure 39 It is a diagram showing an example of a UD diagram with a restoration mark added.

[0051] Figure 40 It is a diagram showing an example of a second parallax image.

[0052] Figure 41 This is a diagram showing an example of a second parallax image.

[0053] Figure 42 This is a diagram showing Figure 41 the first image corresponding to the second parallax image shown.

[0054] Figure 43 This is a diagram showing Figure 40 the parallax pixels used for determining the height of an object in the second parallax image shown.

[0055] Figure 44 This is a diagram showing Figure 41 the parallax pixels used for determining the height of an object in the second parallax image shown.

[0056] Figure 45 This is a flowchart showing an example of the determination process of representative parallax.

[0057] Figure 46 This shows an example of the representative parallax corresponding to the u direction obtained.

[0058] Figure 47 This shows an example of the averaged representative parallax corresponding to the u direction.

[0059] Figure 48 This is a diagram showing an example of the distribution in the UD diagram of the point group representing the representative parallax.

[0060] Figure 49 This is a diagram of observing a road surface from the height direction (y direction).

[0061] Figure 50 This is a diagram transformed into a point group on the x - z plane of the actual space representing the representative parallax.

[0062] Figure 51 This is a diagram showing an example of the output method of the detection result of an object. Detailed implementation mode

[0063] In existing object detection devices, it is necessary to improve the performance of detecting objects. The object detection device, object detection system, moving body, and object detection method of the present invention can improve the performance of detecting objects.

[0064] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. In the following drawings, the same or similar components are given the same reference numerals. In addition, the drawings used in the following description are schematic drawings. The dimensions, ratios, etc. on the drawings do not necessarily match the actual objects. The drawings showing the captured images and parallax images captured by the camera include the drawings created for illustration. These images are different from the actually captured or processed images. In addition, in the following description, the "object to be photographed" is the object photographed by the camera. The "object to be photographed" includes objects, road surfaces, the sky, etc. An "object" is an object having a specific position and size in space. An "object" is also referred to as a "three-dimensional object".

[0065] As Figure 1 shown, the object detection system 1 includes a stereo camera 10 and an object detection device 20. The stereo camera 10 and the object detection device 20 can communicate with each other via wired or wireless communication. The stereo camera 10 and the object detection device 20 can communicate via a network. The network can include, for example, a wired or wireless LAN (Local Area Network), or a CAN (Controller Area Network), etc. The stereo camera 10 and the object detection device 20 can be housed in the same housing and configured integrally. The stereo camera 10 and the object detection device 20 can be configured to be located inside a moving body 30 described later and can communicate with an ECU (Electronic Control Unit) inside the moving body 30.

[0066] In the present invention, a "stereo camera" is a plurality of cameras that have parallax with each other and cooperate with each other. The stereo camera includes at least two or more cameras. In the stereo camera, a plurality of cameras can cooperate to photograph an object from multiple directions. The stereo camera can be a device that includes a plurality of cameras in one housing. The stereo camera can be a device that includes two or more cameras that are independent of each other and separated from each other. The stereo camera is not limited to a plurality of independent cameras. In the present invention, for example, a camera having an optical mechanism that guides light incident on two separate parts to one light receiving element can be used as the stereo camera. In the present invention, multiple images taken of the same object to be photographed from different viewpoints are sometimes referred to as "stereo images".

[0067] As Figure 1As shown, the stereo camera 10 has a first camera 11 and a second camera 12. The first camera 11 and the second camera 12 each have an optical system and an imaging element that define the optical axis OX. The first camera 11 and the second camera 12 each have a different optical axis OX. In the present embodiment, only a single reference numeral OX is used to collectively denote the optical axes OX of both the first camera 11 and the second camera 12. The imaging element includes a CCD image sensor (Charge-Coupled Device Image Sensor) and a CMOS image sensor (Complementary MOS Image Sensor). The imaging elements respectively included in the first camera 11 and the second camera 12 may be located in the same plane perpendicular to the optical axis OX of their respective cameras. The first camera 11 and the second camera 12 generate image signals representing the images formed by the imaging elements. In addition, the first camera 11 and the second camera 12 may perform any processing such as distortion correction, brightness adjustment, contrast adjustment, and gamma correction on the captured images.

[0068] The optical axes OX of the first camera 11 and the second camera 12 are oriented in directions such that they can each capture the same subject. The optical axes OX and positions of the first camera 11 and the second camera 12 are determined such that at least the same subject is included in the captured images. The optical axes OX of the first camera 11 and the second camera 12 are oriented parallel to each other. This parallelism is not limited to strict parallelism, and assembly deviations, mounting deviations, and deviations over time are allowed. The optical axes OX of the first camera 11 and the second camera 12 are not limited to being parallel and may be oriented in different directions from each other. Even when the optical axes OX of the first camera 11 and the second camera 12 are not parallel to each other, a stereo image can be generated by transforming the images within the stereo camera 10 or the object detection device 20. The baseline length is the distance between the optical center of the first camera 11 and the optical center of the second camera 12. The baseline length corresponds to the distance between the centers of the lenses between the first camera 11 and the second camera 12. The baseline length direction is the direction connecting the optical center of the first camera 11 and the optical center of the second camera 12.

[0069] The first camera 11 and the second camera 12 are arranged separately in a direction intersecting the optical axis OX. In one of the multiple embodiments, the first camera 11 and the second camera 12 are arranged along the left - right direction. When facing forward, the first camera 11 is located on the left side of the second camera 12. When facing forward, the second camera 12 is located on the right side of the first camera 11. The first camera 11 and the second camera 12 capture the subject at a specified frame rate (e.g., 30 fps). Due to the difference in the positions of the first camera 11 and the second camera 12, the positions of the corresponding subjects in the two images captured by each camera are different. The first camera 11 captures the first image. The second camera 12 captures the second image. The first image and the second image are stereoscopic images captured from different viewpoints.

[0070] As Figure 2 and Figure 3 shown, the object detection system 1 is mounted on the moving body 30. As Figure 2 shown, the first camera 11 and the second camera 12 are configured such that the optical axes OX of their respective optical systems are substantially parallel to the front of the moving body 30 so as to be able to capture the front of the moving body 30.

[0071] The moving body 30 of the present invention travels on a traveling road including a road or a runway, etc. The surface of the traveling road on which the moving body 30 travels is also referred to as the "road surface".

[0072] In the present invention, the traveling direction when the moving body 30 travels straight is also referred to as the "front" or the "positive direction of the z - axis". The direction opposite to the front is also referred to as the "rear" or the "negative direction of the z - axis". When not particularly distinguishing between the positive direction and the negative direction of the z - axis, these are collectively referred to as the "z - direction". Based on the state where the moving body 30 faces forward, the left - hand direction and the right - hand direction are defined. The z - direction is also referred to as the "depth direction".

[0073] In the present invention, the direction orthogonal to the z - direction and from the left - hand direction to the right - hand direction is also referred to as the "positive direction of the x - axis". The direction orthogonal to the z - direction and from the right - hand direction to the left - hand direction is also referred to as the "negative direction of the x - axis". When not particularly distinguishing between the positive direction and the negative direction of the x - axis, these are collectively referred to as the "x - direction". The x - direction can coincide with the baseline length direction. The x - direction is also referred to as the "horizontal direction".

[0074] In the present invention, the direction perpendicular to the road surface near the moving body 30 and upward from the road surface is also referred to as the "height direction" or "the positive direction of the y-axis". The direction opposite to the height direction is also referred to as the "negative direction of the y-axis". When not particularly distinguishing between the positive direction and the negative direction of the y-axis, these are collectively referred to as the "y-direction". The y-direction can be orthogonal to the x-direction and the z-direction. The y-direction is also referred to as the "vertical direction".

[0075] The "moving body" in the present invention can include, for example, vehicles and aircraft. Vehicles can include, for example, automobiles, industrial vehicles, railway vehicles, living vehicles, and fixed-wing aircraft traveling on a runway, etc. Automobiles can include, for example, sedans, trucks, buses, two-wheel vehicles, and trolleybuses, etc. Industrial vehicles can include, for example, industrial vehicles for agriculture and construction, etc. Industrial vehicles can include, for example, forklifts and golf carts, etc. Industrial vehicles for agriculture can include, for example, tractors, cultivators, transplanters, binders, combine harvesters, and lawn mowers, etc. Industrial vehicles for construction can include, for example, bulldozers, scrapers, excavators, cranes, dump trucks, and loaders, etc. Vehicles can include vehicles that are driven manually. The classification of vehicles is not limited to the above examples. For example, automobiles can include industrial vehicles that can travel on roads. The same vehicle can be included in multiple classifications. Aircraft can include, for example, fixed-wing aircraft and rotary-wing aircraft, etc.

[0076] The first camera 11 and the second camera 12 can be mounted at various positions of the moving body 30. In one embodiment among multiple embodiments, the first camera 11 and the second camera 12 can be mounted inside the moving body 30 which is a vehicle, and can photograph the outside of the moving body 30 through the windshield. For example, the first camera 11 and the second camera 12 are arranged in front of the in-vehicle rearview mirror or on the instrument panel. In one embodiment among multiple embodiments, the first camera 11 and the second camera 12 can be fixed to any one of the front bumper, fender grille, side fender, lamp module, and hood of the vehicle.

[0077] The object detection device 20 can be located at any part inside the moving body 30. For example, the object detection device 20 can be located inside the instrument panel of the moving body 30. The object detection device 20 acquires the first image and the second image from the stereo camera 10. The object detection device 20 detects an object based on the first image and the second image. When the moving body 30 is a vehicle, the object to be detected by the object detection device 20 can be an object on the road surface. As an example of such an object on the road surface, other vehicles and pedestrians, etc. can be cited.

[0078] The object detection device 20 can be configured to read a program recorded in a non-transitory computer-readable medium to implement the processes executed by the control unit 24 described below. The non-transitory computer-readable medium includes, but is not limited to, magnetic storage media, optical storage media, magneto-optical storage media, and semiconductor storage media. Magnetic storage media include magnetic disks, hard disks, and magnetic tapes. Optical storage media include optical discs such as CD (Compact Disc), DVD, and Blu-ray (registered trademark) Disc. Semiconductor storage media include ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), and flash memory.

[0079] The object detection device 20 includes an acquisition unit 21, an output unit 22, a memory 23, and a control unit 24 (processor).

[0080] The acquisition unit 21 is an input interface of the object detection device 20. The acquisition unit 21 can receive input of information from the stereo camera 10 and other devices. In the acquisition unit 21, a physical connector and a wireless communication device can be adopted. The physical connector includes: an electrical connector corresponding to transmission using an electrical signal, an optical connector corresponding to transmission using an optical signal, and an electromagnetic connector corresponding to transmission using an electromagnetic wave. The electrical connector includes: a connector conforming to IEC60603, a connector conforming to the USB standard, a connector corresponding to an RCA terminal, a connector corresponding to an S terminal specified by EIAJ CP-1211A, a connector corresponding to a D terminal specified by EIAJ RC-5237, a connector conforming to the HDMI (registered trademark) standard, and a connector corresponding to a coaxial cable including BNC. The optical connector includes various connectors conforming to IEC 61754. The wireless communication device includes: Bluetooth (registered trademark) and wireless communication devices conforming to various standards including IEEE802.11. The wireless communication device includes at least one antenna.

[0081] Image data of images respectively captured by the first camera 11 and the second camera 12 can be input to the acquisition unit 21. The acquisition unit 21 outputs the input image data to the control unit 24. The acquisition unit 21 can correspond to the transmission method of the shooting signal of the stereo camera 10. The acquisition unit 21 can be connected to the output interface of the stereo camera 10 via a network.

[0082] The output unit 22 is the output interface of the object detection device 20. The output unit 22 can output the processing result of the object detection device 20 to other devices within the mobile body 30 or other devices outside the mobile body 30. Other devices within the mobile body 30 may include: driving assistance devices such as automatic cruise control and safety devices such as automatic braking devices. Other devices outside the mobile body 30 may include other vehicles and roadside detectors, etc. Other devices within the mobile body 30 or other devices outside the mobile body 30 can appropriately use the information received from the object detection device 20. Similar to the acquisition unit 21, the output unit 22 may include various interfaces corresponding to wired and wireless communications. For example, the output unit 22 may have a CAN interface. In this case, the output unit 22 communicates with other devices within the mobile body 30.

[0083] The memory 23 stores programs for various processes and information during operation. The memory 23 includes: volatile memory and non-volatile memory. The memory 23 includes a memory independent of the processor and an on-chip memory of the processor.

[0084] The control unit 24 includes one or more processors. Processors include: general-purpose processors that read specific programs and execute specific functions and dedicated processors for specific processes. Dedicated processors include application-specific integrated circuits (ASICs). Processors include programmable logic devices (PLDs). PLDs include field-programmable gate arrays (FPGAs). The control unit 24 can be either a system-on-a-chip (SoC) or a system-in-a-package (SiP) in which one or more processors cooperate. The processing performed by the control unit 24 can also be described as the processing performed by the processor.

[0085] In the information processing within the object detection device 20, the control unit 24 performs various processes on the first disparity map. The first disparity map is a map that associates two-dimensional coordinates with disparities. The two-dimensional coordinates of the first disparity map are composed of a horizontal direction corresponding to the horizontal direction of the captured image of the stereo camera 10 and a vertical direction intersecting the horizontal direction. The horizontal direction is the first direction. The vertical direction is the second direction. The horizontal direction and the vertical direction may be orthogonal to each other. The horizontal direction may correspond to the width direction of the road surface. When the captured image of the stereo camera 10 includes a horizontal line, the horizontal direction may correspond to the direction parallel to the horizontal line. The vertical direction may correspond to the direction in which gravity is applied in the actual space.

[0086] In the information processing within the object detection device 20, the first disparity map undergoes various operations. The various operations include: arithmetic processing, writing to the memory 23, and reading from the memory 23, etc. The image obtained by visualizing the first disparity map is also referred to as the "first disparity image". The first disparity image is an image in which pixels representing disparity are arranged on a two-dimensional plane composed of horizontal and vertical directions. Hereinafter, the case where the control unit 24 performs various processes on the first disparity image will be described. In the following description, the processing of the first disparity image can be equivalently described as the processing of the first disparity map.

[0087] The structure of the object detection system 1 of the present invention is not limited to Figure 1 the structure shown. Figure 4 The object detection system 1A showing another embodiment of the present invention is shown. The object detection system 1A includes: a stereo camera 10, an object detection device 20, and a generation device 25. The generation device 25 can be mounted on hardware separate from the object detection device 20. The generation device 25 generates a first disparity image based on the first image and the second image output from the stereo camera 10. The generation device 25 has a processor. The processor included in the generation device 25 generates a first disparity image based on the first image and the second image output from the first camera 11 and the second camera 12 of the stereo camera 10, respectively. The acquisition unit 21 acquires the first disparity image from the generation device 25. The object detection device 20 and the generation device 25 can be regarded as a single object detection device.

[0088] Hereinafter, with reference to Figure 5 the flowchart shown, the processing performed by the control unit 24 will be described. Figure 5 is a flowchart showing Figure 1 an example of the overall flow of the processing performed by the object detection device 20 shown.

[0089] First, before detailing the processing performed in each step of the Figure 5 flowchart, a brief description of the outline and purpose of the processing in each step will be given.

[0090] Step S101 is a step of acquiring or generating a first disparity image. Step S101 corresponds to the pre-stage processing of the first processing described later. In the Figure 1 structure shown, the control unit 24 generates a first disparity image. In the Figure 4 structure shown, the control unit 24 acquires the first disparity image generated by the generation device 25 through the acquisition unit 21.

[0091] Step S102 is a step of estimating the shape of the road surface. The process performed in step S102 is also referred to as the "first process". By estimating the shape of the road surface, it is possible to estimate, on the first parallax image, the parallax corresponding to the road surface with respect to the coordinates in the longitudinal direction. The shape of the road surface can be used to remove unnecessary parallax and / or estimate the height position of the road surface in the actual space in the following processes.

[0092] Step S103 is a step of generating a second parallax image by removing unnecessary parallax from the first parallax image. The process performed in step S103 is also referred to as the "second process". Unnecessary parallax includes, for example, parallax corresponding to the white lines of the road surface and parallax corresponding to buildings existing above the road that may be included in the first parallax image. By removing unnecessary parallax from the first parallax image, the possibility that the object detection device 20 misdetects the white lines of the road surface and buildings existing above the road surface as objects to be detected on the road surface is reduced. Thereby, the accuracy of the object detection device 20 in detecting objects can be improved.

[0093] Step S103 is a step of generating a second parallax map by removing unnecessary parallax from the first parallax map in the information processing within the object detection device 20. The above-mentioned second parallax image is an image obtained by imaging the second parallax map. In the following description, the processing of the second parallax image can also be referred to as the processing of the second parallax map.

[0094] Step S104 is a step of detecting the first parallax and the second parallax based on the second parallax image. The first parallax is detected by regarding it as the parallax corresponding to the object to be detected. The second parallax is detected by regarding it as a candidate for the parallax corresponding to the object to be detected. It can be determined whether to restore the second parallax as the parallax corresponding to the object to be detected in the subsequent restoration process.

[0095] Step S105 is a step of calculating the height of the object in the image. The height of the object detected in the image can be the height of the detection frame 182 as shown below. The height of the object in the image can also be referred to as the "height of the detection frame". The process performed in step S105 is also referred to as the "height calculation process". Figure 51 shown

[0096] Step S106 is a step of detecting a parallax corresponding to an object parallel to the traveling direction of the moving body 30 in the second parallax image. An object parallel to the traveling direction of the moving body 30 is also referred to as a "parallel object". As an example of a parallel object, roadside buildings such as guardrails and sound insulation walls of highways or the sides of other vehicles can be cited. For example, on the second parallax image, parallel objects such as guardrails and objects to be detected such as other vehicles may be close. By detecting parallel objects, for example, adding a mark to the detected parallel objects, the possibility that the object detection device 20 misdetects the parallel objects as objects to be detected on the road surface is reduced. With such a structure, the accuracy of the object detection device 20 in detecting objects can be improved. The process executed in step S106 is also referred to as "detection process of parallel objects".

[0097] Step S107 is a step of determining whether to restore the second parallax to the parallax corresponding to the object to be detected.

[0098] Step S108 is a step of determining a representative parallax from the first parallax and the restored second parallax in each coordinate in the horizontal direction of the second parallax image. The process executed in step S108 is also referred to as "third process". The processes executed in steps S104 to S108 are also referred to as "third process".

[0099] Step S109 is a step of detecting an object by transforming the information of the representative parallax into coordinates in the actual space and extracting a set (group) of representative parallaxes. The process executed in step S109 is also referred to as "fourth process". In step S109, information on the position of the object to be detected and the width of the object observed from the side of the stereo camera 10 is obtained.

[0100] Step S110 is a step of outputting the information of the detected object from the output unit 22. The information of the object output in step S110 may include: the height of the object on the image calculated in step S105, the position of the object to be detected detected in step S109, and the information on the width of the object observed from the side of the stereo camera 10 detected in step S109, etc. This information can be provided to other devices in the moving body 30.

[0101] Next, each step will be described in detail.

[0102] In the process of step S101, the control unit 24 acquires or generates a first parallax image. In Figure 1 the object detection system 1 shown, the control unit 24 generates a first parallax image based on the first image and the second image acquired by the acquisition unit 21. In Figure 4In the object detection system 1A shown, the control unit 24 acquires, via the acquisition unit 21, a first disparity image generated by the generation device 25. The control unit 24 may store the first disparity image in the memory 23 for subsequent processing.

[0103] A method for generating the first disparity image is well-known, and thus will be briefly described below. Hereinafter, it is assumed that the control unit 24 generates the first disparity image.

[0104] The control unit 24 acquires a first image captured by the first camera 11 and a second image captured by the second camera 12. The control unit 24 divides one of the first image and the second image (for example, the first image) into a plurality of small regions. The small region may be a rectangular region in which a plurality of pixels are arranged in the vertical and horizontal directions, respectively. For example, the small region may be composed of three pixels arranged vertically and three pixels arranged horizontally. However, the number of pixels arranged vertically in the small region and the number of pixels arranged horizontally are not limited to three. In addition, the number of pixels included in the vertical and horizontal directions of the small region may be different. The control unit 24 performs matching by shifting each pixel of the divided plurality of small regions horizontally one by one on the other image and comparing the feature amounts. For example, when the first image is divided into small regions, the control unit 24 shifts the small regions of the first image horizontally one by one on the second image and compares the feature amounts, thereby performing matching. The feature amount is, for example, a pattern of brightness and color. In the matching of stereo images, a method using the SAD (Sum of Absolute Differences) function is known. Here, it represents the sum of the absolute values of the differences in the brightness values within the small region. When the SAD function is minimized, it is determined that the two images are the most similar. The matching of stereo images is not limited to the method using the SAD function. Other methods may also be employed in the matching of stereo images.

[0105] The control unit 24 calculates the disparity of each small region based on the difference in the horizontal pixel positions of the two regions that are matched in the first image and the second image. The disparity may be the difference between the position in the first image and the position in the second image of the same object being photographed. The magnitude of the disparity may be expressed in units of the horizontal width of a pixel in the stereo image. By performing interpolation processing, the magnitude of the disparity can be calculated with a precision less than one pixel. The magnitude of the disparity corresponds to the distance between the object being photographed by the stereo camera 10 and the stereo camera 10 in the actual space. The closer the distance from the stereo camera 10 to the object being photographed in the actual space, the larger the disparity corresponding to the object being photographed. The farther the distance from the stereo camera 10 to the object being photographed in the actual space, the smaller the disparity corresponding to the object being photographed.

[0106] The control unit 24 generates a first disparity image representing the distribution of the calculated disparities. The pixels representing the disparities that make up the first disparity image are also referred to as "disparity pixels". The control unit 24 can generate the first disparity image with the same clarity as the pixels of the original first image and second image.

[0107] Figure 6 The first disparity image 40 is shown in . The first disparity image 40 is a two-dimensional plane formed by the horizontal direction (first direction) and the vertical direction (second direction) orthogonal to the horizontal direction of the stereo camera 10. Disparity pixels representing disparities are arranged in the two-dimensional plane of the first disparity image 40. The horizontal direction is also referred to as the "u direction". The vertical direction is also referred to as the "v direction". The coordinate system formed by the u direction and the v direction is also referred to as the "uv coordinate system" and the "image coordinate system". In the present embodiment, the upper left corner of each figure facing the paper surface is the origin (0, 0) of the uv coordinate system. In addition, the direction from the left side to the right side of each figure is the positive direction of the u axis, and the opposite direction is the negative direction of the u axis. In addition, the direction from the upper side to the lower side of each figure is the positive direction of the v axis, and the opposite direction is the negative direction of the v axis. The negative direction of the v axis corresponds to the direction from the road surface upward in the actual space. The u coordinate and the v coordinate can be represented in units of disparity pixels.

[0108] As Figure 6 shown, the first disparity image 40 includes a disparity image 41, a disparity image 42, and a disparity image 43. The disparity image 41 corresponds to the road surface in front of the moving body 30. The disparity image 42 corresponds to other vehicles located in front of the moving body 30. The disparity image 43 is a disparity image corresponding to the guardrail.

[0109] The control unit 24 can display the disparity information of each pixel including the first disparity image by the brightness or color of each pixel, etc. In Figure 6 the first disparity image 40 shown, for the sake of easy explanation, the disparities of each pixel are shown by different shades. In the first disparity image 40, the darker the shade of a region, the smaller the disparity represented by the pixels in that region. The thinner the shade of a region, the larger the disparity represented by the pixels in that region. In the first disparity image 40, the pixels within a region of equal shade all represent disparities within a specified range. In an actual first disparity image, since the pixels in some regions have fewer feature amounts on the stereo image in the above-mentioned matching process for calculating disparities compared to the pixels in other regions, it is sometimes difficult to calculate the disparities. For example, it is difficult to calculate the disparities for parts of spatially uniform objects such as vehicle windows and parts where highlight overflow (whitening) caused by sunlight reflection occurs. In the first disparity image, when there are disparities corresponding to objects and buildings, they can be represented by different brightnesses or colors from the disparities corresponding to the background located farther away.

[0110] The control unit 24 may not display the first parallax image as an image after calculating the parallax. That is, the control unit 24 only needs to hold the first parallax map that is the basis of the first parallax image and perform appropriate processing on the first parallax map.

[0111] After executing step S101, the control unit 24 executes a first process (step S102) of estimating the shape of the road surface based on the first parallax image. Hereinafter, the flowchart of Figure 7 、 Figure 8 and Figure 14 is used to illustrate the estimation process of the shape of the road surface executed by the control unit 24. First, the control unit 24 extracts a road surface candidate parallax d c (step S201). The road surface candidate parallax d c is a parallax with a high possibility of matching the road surface parallax d r collected from the first parallax image. The road surface parallax d r refers to the parallax of the road surface area. The road surface parallax d r does not include the parallax of the objects on the road surface. The road surface parallax d r represents the distance to the corresponding part on the road surface. The road surface parallax d r is collected as a value having a value close to each other at positions having the same v coordinate.

[0112] In Figure 8 the flowchart shows the specific content of the extraction process of the road surface candidate parallax d c . As Figure 8 shown, the control unit 24 calculates an initial value of the parallax for calculating the road surface candidate parallax, that is, a road surface candidate parallax initial value d0 (step S301) based on the installation position of the stereo camera 10. The road surface candidate parallax initial value d0 is the initial value of the road surface candidate parallax at the extraction position of the road surface candidate parallax closest to the stereo camera 10. The extraction position of the road surface candidate parallax closest to the stereo camera 10 can be set, for example, in the range from 1 m to 10 m from the stereo camera.

[0113] As Figure 9As shown, in the stereo camera 10, the road surface height Y is the height of the stereo camera 10 in the vertical direction from the photographed road surface 41A. In addition, the road surface height Y0 is the height of the installation position of the stereo camera 10 from the road surface 41A. Due to the undulation of the road, the road surface height Y may change depending on the distance from the stereo camera 10. Therefore, the road surface height Y at a position far from the stereo camera 10 is not the same as the road surface height Y0 at the installation position of the stereo camera 10. In one embodiment of the multiple embodiments, it is assumed that the first camera 11 and the second camera 12 of the stereo camera 10 are arranged so that the optical axes OX are parallel to each other and face forward. In Figure 9 , the distance Z represents the horizontal distance to a specific road surface position. Let the baseline length of the stereo camera 10 be B and the vertical image size be TOTALv. In this case, the road surface parallax d s of the road surface 41A photographed at a certain vertical coordinate (v coordinate) has the following relationship with the road surface height Y.

[0114] d s = B / Y×(v - TOTALv / 2) (1)

[0115] The road surface parallax d s calculated by the mathematical formula (1) s is also referred to as the "geometrically estimated road surface parallax". Hereinafter, the geometrically estimated road surface parallax is sometimes represented by the reference numeral d

[0116] The initial value d0 of the road surface candidate parallax is a value calculated on the assumption that the road surface 41A is parallel and flat to the optical axes OX of the first camera 11 and the second camera 12 between the stereo camera 10 and the extraction position of the road surface candidate parallax d c at the position closest to the stereo camera 10. In this case, the v coordinate of the extraction position of the road surface candidate parallax at the position closest to the stereo camera 10 on the first parallax image is determined as a specific coordinate (v0). The coordinate (v0) is the initial value of the v coordinate for extracting the road surface candidate parallax. The coordinate (v0) is between TOTALv / 2 and TOTALv. The coordinate (v0) is located at the lowermost side (the side with the larger v coordinate) within the range of the image coordinates where the parallax can be calculated. The coordinate (v0) can be TOTALv corresponding to the lowermost row of the first parallax image. The initial value d0 of the road surface candidate parallax can be determined by substituting v0 into v in the mathematical formula (1) and substituting Y0 into Y.

[0117] The control unit 24 calculates the parallax collection threshold for the v - coordinate in the longitudinal direction, which is the coordinate (v0), i.e., the first row, based on the initial road surface candidate parallax d0 (step S302). A row refers to an array of pixels arranged horizontally with the same v - coordinate on the first parallax image. The parallax collection threshold includes an upper threshold, which is the threshold for the upper limit of collecting parallax, and a lower threshold, which is the threshold for the lower limit of collecting parallax. Based on a specified rule, the parallax collection threshold is set above and below the initial road surface candidate parallax d0 so that the parallax collection threshold includes the initial road surface candidate parallax d0. Specifically, the road surface parallax when the road surface height Y is changed by a predetermined road surface height change amount ΔY up and down from the state where the initial road surface candidate parallax d0 is calculated is determined as the upper threshold and the lower threshold of the parallax collection threshold. That is, the lower threshold of the parallax collection threshold is obtained by subtracting the parallax corresponding to the road surface height change amount ΔY from the initial road surface candidate parallax d0. The upper threshold of the parallax collection threshold is obtained by adding the parallax corresponding to the road surface height change amount ΔY to the initial road surface candidate parallax d0. The specific lower threshold and upper threshold of the parallax collection threshold are obtained by changing the value of Y in the mathematical formula (1).

[0118] Hereinafter, the control unit 24 repeatedly executes the processes of steps S303 to S307. First, the control unit 24 processes the row with the v - coordinate of (v0) located at the lowermost side of the first parallax image (step S303).

[0119] The control unit 24 collects parallax using the parallax collection threshold (step S304). The control unit 24 collects, as the road surface candidate parallax d, the parallax pixels having a parallax between the lower threshold and the upper threshold of the parallax collection threshold for each parallax pixel whose v - coordinate in the first parallax image is positioned side - by - side horizontally along the coordinate (v0). c That is, the control unit 24 determines, as candidates for parallax pixels representing the correct parallax of the road surface 41A, the parallax pixels having a parallax within a specified range bounded by the initial road surface candidate parallax d0 calculated using the mathematical formula (1). The control unit 24 sets the parallax of the parallax pixels determined as candidates for parallax pixels representing the correct parallax of the road surface 41A as the road surface candidate parallax d c With such a structure, the control unit 24 can reduce the possibility of misjudging the parallax corresponding to an object other than the road surface 41A, such as an object on the road surface 41A and a structure, as the parallax corresponding to the road surface 41A. As a result, the detection accuracy of the road surface 41A is improved.

[0120] In the process of step S304, when the determination of all parallax pixels with the v - coordinate of (v0) is completed, the control unit 24 averages the collected road surface candidate parallax d c to calculate the road surface candidate parallax d cThe average value is the average road surface candidate disparity d av (Step S305). The control unit 24 can calculate each road surface candidate disparity d c along with its u-v coordinates, and the average road surface candidate disparity d av at the v coordinate (v0), and store them in the memory 23.

[0121] After performing the process of step S305, the control unit 24 performs the process of step S306. In the process of step S306, for each disparity pixel in the row where the v coordinate is (v0 - 1), the control unit 24 calculates a disparity collection threshold based on the average road surface candidate disparity d av at the v coordinate (v0) calculated in the process of step S305. The control unit 24 changes the road surface height Y for the average road surface candidate disparity d av at the v coordinate (v0) calculated in the process of step S305 to make the mathematical formula (1) hold. The control unit 24 calculates the geometrically estimated road surface disparity d s at the v coordinate (v0 - 1) by substituting (v0 - 1) in place of v0 in the mathematical formula (1) with the changed road surface height Y. The control unit 24 can set the disparity obtained by subtracting a specified road surface height change amount ΔY from the geometrically estimated road surface disparity d s as the lower threshold of the disparity collection threshold in a manner similar to the process of step S302. The control unit 24 can set the disparity obtained by adding the geometrically estimated road surface disparity d s and the disparity of the specified road surface height change amount ΔY as the upper threshold of the disparity collection threshold.

[0122] After performing the process of step S306, the control unit 24 determines whether the geometrically estimated road surface disparity d s calculated by the mathematical formula (1) is greater than a specified value. The specified value is, for example, 1 pixel. When the geometrically estimated road surface disparity d s is greater than 1, the control unit 24 returns to the process of step S303 (step S307). In the process of step S303, the control unit 24 moves the extraction object of the road surface candidate disparity d c to the row one pixel above. That is, when the extraction object of the road surface candidate disparity d c is at the row with the v coordinate (v0), the control unit 24 changes the v coordinate of the row of the road surface detection object to (v0 - 1). Additionally, as Figure 10 shown, when the calculation object of the road surface candidate disparity d c is the nth row, the control unit 24 changes the row of the road surface detection object to the (n + 1)th row. For illustration, in Figure 10The vertical width of each row is expanded. The actual height of each row is the height of 1 pixel. In this case, the v coordinate of the (n + 1)-th row is 1 less than the v coordinate of the n-th row.

[0123] The processes of steps S304 to S306 for the (n + 1)-th row are performed in the same or similar manner as the processes for the row with the v coordinate (v0). In the process of step S304, the control unit 24 collects the road surface candidate disparity d using the disparity collection threshold calculated in the process of step S306 for the n-th row. c In the process of step S305, the control unit 24 c averages the collected road surface candidate disparities d to calculate the average road surface candidate disparity d. av In the process of step S306, the control unit 24 av uses the average road surface candidate disparity d to change the road surface height Y in the mathematical formula (1). The control unit 24 uses the mathematical formula (1) with the changed road surface height Y to calculate the geometrically estimated road surface disparity d. s Furthermore, the control unit 24 calculates the disparity collection threshold by considering the road surface height change amount ΔY in the geometrically estimated road surface disparity d to extract the road surface candidate disparity d for the (n + 2)-th row. s c

[0124] The control unit 24 extracts the road surface candidate disparity d corresponding to the v coordinate while moving the extraction target of the road surface candidate disparity d c sequentially upward (negative direction of the v coordinate) from the row corresponding to the extraction position of the road surface candidate disparity d closest to the stereo camera 10. The control unit 24 may store the extracted road surface candidate disparity d c together with the corresponding u coordinate, v coordinate, and the average road surface candidate disparity d corresponding to the v coordinate in the memory 23. c c av

[0125] In the process of step S307, when the geometrically estimated road surface disparity d calculated by the mathematical formula (1) s is equal to or less than the above-mentioned specified value, the control unit 24 ends the extraction process of the road surface candidate disparity d and returns to c Figure 7 the process of step S201 in the flowchart. The specified value may be, for example, 1 pixel.

[0126] In this way, in the Figure 8 flowchart, when observing from the stereo camera 10, the road surface candidate disparity d is extracted. cThe initial value of the v coordinate is set to v0 corresponding to the position on the near side, and the road surface candidate disparity d on the far side is sequentially extracted. c Generally, compared with the far side, the stereo camera 10 has higher detection accuracy for detecting disparities on the near side. Therefore, by sequentially extracting the road surface candidate disparity d from the near side to the far side c , the accuracy of the detected road surface candidate disparity d c can be improved.

[0127] In the flowchart for extracting the above road surface candidate disparity d c of Figure 8 , the road surface candidate disparity d is calculated for each coordinate in the vertical direction. c In other words, in the flowchart for extracting the above road surface candidate disparity d c , the road surface candidate disparity d is calculated for each row of pixels in the vertical direction. c The unit for calculating the road surface candidate disparity is not limited to this. The road surface candidate disparity d can also be calculated by aggregating multiple coordinates in the vertical direction. c .

[0128] Continuing the extraction process of the road surface candidate disparity d in steps S301 to S307, the control unit 24 proceeds to the process of step S202 in the c flowchart. When sequentially estimating the road surface disparity d from the near side to the far side Figure 7 , the control unit 24 sequentially applies the Kalman filter to the road surface disparity d r . Therefore, the control unit 24 first initializes the Kalman filter (step S202). The average road surface candidate disparity d r corresponding to the bottommost row (the row with the v coordinate value of v0) in the row where the road surface disparity d is estimated in the process of step S305 can be used as the initial value of the Kalman filter. r av

[0129] The control unit 24 sequentially executes the following processes of steps S203 to S210 (step S203) while changing the row to be the object from the near side to the far side of the road surface.

[0130] First, for the row to be the object in the first disparity image, the control unit 24 generates a histogram representing the frequency of each value of the road surface disparity d c based on the road surface candidate disparity d within a range of a constant width in the actual space. The range of a constant width in the actual space is a range considering the width of the driving lane of the road. The constant width can be set to a value such as 2.5 m or 3.5 m, for example. The range for obtaining the disparity is, for example, in r Figure 11is initially set to the range surrounded by the solid-line frame line 45. The constant width is pre-stored in the memory 23 of the object detection device 20 or the like. By limiting the range for obtaining the parallax to this range, the possibility that the control unit 24 misidentifies an object other than the road surface 41A or a building such as a sound insulation wall as the road surface 41A can be reduced. Thereby, the accuracy of road surface detection can be improved. As described later, the range for obtaining the parallax as shown by the solid line in Figure 11 can be sequentially changed according to the conditions on the road ahead.

[0131] For the target row, the control unit 24 sets the acquisition range of the road surface parallax d r based on the predicted value of the road surface parallax d r through the Kalman filter. The acquisition range of the road surface parallax d r is a range determined based on the reliability calculated when predicting the road surface parallax d r of the next row by the Kalman filter. The reliability is represented by the variance σ 2 (σ is the standard deviation of the road surface parallax d r ). The control unit 24 can obtain the acquisition range of the road surface parallax d r by, for example, predicted value ± 2σ. The control unit 24 extracts the road surface parallax d c with the highest frequency in the acquisition range of the road surface parallax d r set based on the Kalman filter according to the histogram of the road surface candidate parallax d r generated in the process of step S204. The control unit 24 sets the extracted road surface parallax d r as the observed value of the road surface parallax d r of the target row (step S205).

[0132] Next, the control unit 24 confirms that the road surface parallax d r determined in the process of step S205 is the correct road surface parallax d r and does not include parallax corresponding to an object, etc. (step S206). For all the road surface parallax d r detected in each row up to the current process, the control unit 24 generates a d-v correlation diagram mapped onto the d-v coordinate space with the road surface parallax d r and the v coordinate as the coordinate axes. When the road surface 41A is correctly detected, in the d-v correlation diagram, as shown by the dashed line in Figure 12 , the road surface parallax d r decreases linearly as the value of the v coordinate decreases.

[0133] On the other hand, as shown in Figure 13As shown, in the case where the parallax representing an object is misrecognized as the parallax representing the road surface 41A, in the d-v correlation diagram, in the part representing the parallax of the object, the parallax d is substantially constant regardless of the change in the vertical coordinate (v coordinate). Generally, since the object includes a part perpendicular to the road surface 41A, the object is displayed in the first parallax image in such a way that it includes a plurality of equidistant parallaxes. In Figure 13 in the first part R1, the parallax d decreases as the value of the v coordinate changes. The first part R1 is the part where the parallax representing the road surface 41A is correctly detected. In the second part R2, the parallax d is constant even when the v coordinate changes. The second part R2 is considered to be the part where the parallax is misdetected as representing the object. When the rows with substantially equal values of the parallax d continue for a specified number, the control unit 24 can determine that the parallax representing the object is misrecognized as the parallax representing the road surface 41A.

[0134] When it is determined in the process of step S206 that the parallax is not the correct road surface parallax d r (step S206: No), the control unit 24 starts searching for the road surface parallax d again from the row determined to be misdetected as the parallax representing the object r (step S207). In the process of step S207, the control unit 24 searches for the road surface parallax histogram again in the area of the row where the parallax d does not change even when the value of the v coordinate changes. In this area, when there is a relatively high frequency of parallax in the part of the parallax smaller than the parallax d determined in the process of step S205, the control unit 24 can determine that this parallax is the observed value of the correct road surface parallax d r .

[0135] When it is determined in the process of step S206 that the road surface parallax d r is correct (step S206: Yes), or when the re-search for the road surface parallax d r ends in the process of step S207, the control unit 24 proceeds to the process of step S208. In the process of step S208, the control unit 24 determines the horizontal range of the parallax image 41, which corresponds to the road surface on the first parallax image that is the object of generating the histogram for the next row shifted by 1 pixel vertically. For example, as Figure 11 shown, when the parallax image 42 corresponding to another vehicle exists on the parallax image 41 corresponding to the road surface, the control unit 24 cannot obtain the correct road surface parallax d of the parallax image 41 corresponding to the road surface part overlapping with the other vehicle r . When the range of the parallax image 41 corresponding to the road surface for which the road surface parallax d r can be obtained becomes narrow, the control unit 24 has difficulty obtaining the accurate road surface parallax d r . Therefore, as Figure 11As shown by the dashed line in the middle, the control unit 24 causes the range for obtaining the road surface candidate disparity d c to change sequentially in the horizontal direction. Specifically, when it is determined in the process of step S206 that the disparity representing an object is included, the control unit 24 detects on which side in the horizontal direction of the object the correct road surface disparity d r of the road surface candidate disparity d c is more. In the next line, the control unit 24 causes the range for obtaining the disparity to move sequentially to the side ( r in c the right side in Figure 11 the figure) that includes more of the road surface candidate disparity d

[0136] representing the correct road surface disparity d. r Next, the control unit 24 updates the Kalman filter (step S209) using the road surface disparity d r of the current line determined in the process of step S205 or S207. That is, the Kalman filter calculates the estimated value of the road surface disparity d r based on the observed value of the road surface disparity d r of the current line. When the estimated value of the current line is calculated, the control unit 24 adds the estimated value of the road surface disparity d r of the current line as part of the past data and uses it for the process of calculating the estimated value of the road surface disparity d r of the next line. It is considered that the height of the road surface 41A does not change sharply up and down with respect to the horizontal distance Z from the stereo camera 10. Therefore, in the estimation using the Kalman filter of the present embodiment, it is estimated that the road surface disparity d r of the next line exists near the road surface disparity d r of the current line. In this way, by limiting the range of the disparity of the histogram of the next line generated by the control unit 24 near the road surface disparity d

[0137] of the current line, the possibility of erroneously detecting an object other than the road surface 41A is reduced. In addition, the amount of calculation performed by the control unit 24 can be reduced, thereby enabling high-speed processing. r In the process of step S209, when the road surface disparity d r estimated by the Kalman filter is greater than a specified value, the control unit 24 returns to the process of step S203 and repeats the processes of steps S203 to S209. When the road surface disparity d

[0138] In the process of step S211, the control unit 24 approximates the longitudinal image coordinate v and the estimated road surface parallax d using two straight lines on the dv correlation diagram. r The relationship between the road parallax d r The value of the v coordinate is related to the distance Z from the stereo camera 10 and the road surface height Y. Therefore, two straight lines are used to approximate the v coordinate and the road surface parallax d r The relationship between means that the relationship between the distance Z from the stereo camera 10 and the height of the road surface 41A can be approximated by two straight lines. Figure 14 The processing of step S211 is described in detail in the flowchart of FIG.

[0139] First, by going to Figure 7 The process up to step S210 obtains the road parallax d r The correlation between the v coordinate and the road parallax d r The correlation between them is as follows in the dv coordinate space: Figure 15 In the actual space, when the road surface 41A is flat and has no inclination change, the graph 51 is a straight line. However, in the actual road surface 41A, the inclination of the road surface 41A may change due to changes in ups and downs. When the inclination of the road surface 41A changes, the graph 51 in the dv coordinate space cannot be represented by a straight line. If more than three straight lines or curves are used to approximate the change in the inclination of the road surface 41A, the processing load of the object detection device 20 increases. Therefore, in the present application, two straight lines are used to approximate the curve 51.

[0140] like Figure 15 As shown, the control unit 24 uses the first straight line 52 to calculate the estimated road surface parallax d on the lower side (short distance side) in the dv coordinate space based on the least square method. r Approximation is performed (step S401). Approximation using the first straight line 52 can detect a road surface parallax d corresponding to a predetermined distance within the distance range of the object detection target of the object detection device 20. r The predetermined distance can be set to half the distance range of the object detection target of the object detection device 20. For example, when the object detection device 20 is designed to detect an object up to 100 m away, the first straight line 52 can be determined by the least square method as the closest to the figure 51 within the range from the closest distance that can be measured by the stereo camera 10 to 50 m away.

[0141] Next, the control unit 24 determines whether the inclination of the road surface 41A represented by the approximate first straight line 52 in the process of step S401 is an inclination that can possibly exist as the road surface 41A (step S402). The inclination angle of the first straight line 52 becomes a plane when transformed into the real space. The inclination of the first straight line 52 corresponds to the inclination angle of the road surface 41A in the yz plane determined according to conditions such as the road surface height Y0 and the baseline length B at the installation position of the stereo camera 10. When the inclination of the road surface 41A in the real space corresponding to the first straight line 52 is within the range of a specified angle based on the horizontal plane in the real space, the control unit 24 can determine that the inclination of the road surface 41A in the real space corresponding to the first straight line 52 is an inclination that can possibly exist. When the inclination of the road surface 41A in the real space corresponding to the first straight line 52 is outside the range of the specified angle based on the horizontal plane in the real space, the control unit 24 can determine that the inclination of the road surface 41A in the real space corresponding to the first straight line 52 is an inclination that cannot possibly exist. The specified angle can be appropriately set in consideration of the driving environment of the moving body 30.

[0142] In the process of step S402, when it is determined that the inclination of the first straight line 52 is an inclination that cannot possibly exist as the road surface 41A (step S402: No), the control unit 24 determines the first straight line 52 based on a theoretical road surface assuming that the road surface 41A is flat (step S403). The theoretical road surface can be calculated based on the installation conditions such as the road surface height Y0, the installation angle, and the baseline length B at the installation position of the stereo camera 10. When the road surface parallax d r calculated from the image cannot be trusted, the control unit 24 adopts the road surface parallax of the theoretical road surface. For example, when the parallax representing an object or a building other than the road surface 41A is erroneously extracted as the road surface parallax d r the control unit 24 determines that the road surface 41A has an unrealistic inclination and can eliminate the error. Thereby, the possibility of erroneously determining the parallax representing an object or a building other than the road surface 41A as the road surface parallax d r can be reduced.

[0143] In the process of step S402, when the control unit 24 determines that the inclination of the first straight line 52 is an inclination that may exist as the road surface 41A (step S402: Yes), or after the process of step S403 is executed, the control unit 24 proceeds to the process of step S404. In the process of step S404, the control unit 24 determines the approximate start point 53 for starting the approximation of the second straight line 55. The control unit 24 can calculate the approximation error with respect to the figure 51 in the order from the smallest side (far side) to the larger side (near side) of the v coordinate of the first straight line 52, and use the coordinates on the first straight line 52 where the approximation error is continuously less than the specified value as the approximate start point 53. Alternatively, the approximation error with respect to the figure 51 can be calculated in the order from the largest side (near side) to the smaller side (far side) of the v coordinate of the first straight line 52, and the approximate start point 53 can be determined as the coordinates on the first straight line 52 when the approximation error is greater than the specified value. The v coordinate of the approximate start point 53 is not fixed to a specific value. The approximate start point 53 can be set on the first straight line 52 at a position corresponding to a v coordinate closer to the side of the stereo camera 10 than half the distance of the distance range of the object to be detected by the object detection device 20. For example, when the first straight line 52 approximates the road surface 41A in the range from the closest measurable distance to 50 m away, the approximate start point 53 can be set at the position corresponding to the v coordinate of 40 m, which is closer than 50 m.

[0144] After executing the process of step S404, the control unit 24 repeatedly executes the processes of steps S405 to S407. As Figure 16 shown, the control unit 24 uses the angle difference from the first straight line 52 as the angle selected from the specified angle range, and sequentially selects the candidate straight line 54, which is a candidate for the second straight line 55 starting from the approximate start point 53 (step S405). The specified angle range is set to the angle within which the slope of the road can vary within the distance range of the measurement object. The specified angle range can be, for example, ±3 degrees. For example, the control unit 24 can change the angle of the candidate straight line 54 from the angle of the first straight line 52 - 3 degrees and increase it by 0.001 degrees sequentially until the angle of the first straight line 52 + 3 degrees.

[0145] For the selected candidate straight line 54, the control unit 24 calculates the error of the part above (far side) the approximate start point 53 of the figure 51 in the d - v coordinate space (step S406). The calculation of the error can be performed by the mean square error of the parallax d with respect to the v coordinate. The control unit 24 can store the error calculated for each candidate straight line 54 in the memory 23.

[0146] When the calculation of the errors for all candidate straight lines 54 within the angular range is completed (step S407), the control unit 24 searches for the minimum error among the errors stored in the memory 23. As Figure 17 shown, the control unit 24 selects the candidate straight line 54 with the minimum error as the second straight line 55 (step S408).

[0147] When the second straight line 55 is determined in the process of step S408, the control unit 24 determines whether the error of the graphic 51 with respect to the second straight line 55 is within a specified value (step S409). The specified value is appropriately set to obtain the desired accuracy of road surface estimation.

[0148] In the process of step S409, when the error is within the specified value (step S409: Yes), the road surface parallax d is approximated using the first straight line 52 and the second straight line 55 r .

[0149] In the process of step S409, when the error exceeds the specified value (step S409: No), the control unit 24 extends the first straight line 52 upward (the far - distance side) and rewrites the approximation result (step S410). As described above, the road surface parallax d is approximated using two straight lines r .

[0150] The road surface parallax d with respect to the v - coordinate is approximated using two straight lines r , thereby approximating the shape of the road surface using two straight lines. Thus, compared with the case of approximating the shape of the road surface using a curve or three or more straight lines, the load of subsequent calculations can be reduced and the object detection process can be speeded up. In addition, compared with the case of approximating the road surface using a single straight line, the error from the actual road surface is smaller. And since the v - coordinate of the approximation starting point 53 of the second straight line 55 is not fixed to a specified coordinate, compared with the case of pre - fixing the coordinate of the approximation starting point 53, the accuracy of approximation with the actual road surface can be improved.

[0151] In the process of step S409, when the error is within the specified value (step S409: Yes) or after the process of step S410 is executed, the control unit 24 ends the process of approximating the road surface parallax d using straight lines r and returns to Figure 7 the process of step S212.

[0152] In the process of step S212, the threshold value of the road surface parallax d removed from the first parallax image r is determined (step S212). The threshold value of the road surface parallax d removed from the first parallax image r corresponds to the first height described later. The first height can be calculated in such a way that the road surface parallax d r is removed in the process of the next step S103.

[0153] Next, the control unit 24 returns to Figure 5 the flowchart of. Through the above processing, the control unit 24 obtains an approximate expression using two straight lines to approximate the relationship between the v coordinate and the road surface parallax d in the d-v coordinate space r between. According to the approximate expression representing the relationship between the v coordinate and the road surface parallax d r in the d-v coordinate space, the relationship between the distance Z in front of the stereo camera 10 and the road surface height Y in the actual space can be obtained. The control unit 24 performs the second process (step S103) based on the approximate expression. The second process is a process of removing the parallax corresponding to the range with a height of the first height or less from the road surface 41A in the actual space and the parallax corresponding to the object with a height of the second height or more from the road surface 41A in the first parallax image. Thus, the control unit 24 generates Figure 6 the second parallax image 60 as shown in Figure 18 from the first parallax image 40 shown. Figure 18 is a diagram drawn for illustration. The actual second parallax image based on the image obtained from the stereo camera 10 is, for example, as shown in Figure 19 . In Figure 19 , the magnitude of the parallax is represented by the black-and-white shading. Figure 19 The second parallax image shown includes a parallax image 44 corresponding to another vehicle.

[0154] The first height can be set to be less than the minimum value of the height of the object that is the detection target of the object detection device 20. The minimum value of the height of the object that is the detection target of the object detection device 20 can be the height of a child (for example, 50 cm). The first height can be a value greater than 15 cm and less than 50 cm. By including noise in the first parallax image 40 shown in Figure 6 , sometimes the detection accuracy of detecting the parallax corresponding to the road surface by the above-mentioned processing will decrease. In this case, if only the parallax corresponding to the detected road surface is removed from the first parallax image 40, sometimes the parallax corresponding to the road surface will remain in a part of the parallax image 41. By removing the parallax corresponding to the range with a height of the first height or less from the road surface 41A from the first parallax image 40, a second parallax image from which the parallax corresponding to the road surface has been accurately removed from the parallax image 41 can be obtained.

[0155] In Figure 18In the second parallax image 60 shown, information on the parallax corresponding to a range with a height from the road surface in the actual space of the first height or less is removed. With such a configuration, the parallax image 41 corresponding to the road surface does not contain parallax information. The parallax image 41 corresponding to the road surface and the parallax image 42 corresponding to another vehicle are adjacent. In the second parallax image 60, since the parallax image 41 does not contain parallax information, in subsequent processing, the processing of the parallax information of the parallax image 42 corresponding to another vehicle can be made easier. Further, by removing unnecessary parallax that is not related to the object to be detected, the processing speed described later can be increased.

[0156] The second height can be appropriately set based on the maximum value of the height of the object that is the detection target of the object detection device 20. When the moving body 30 is a vehicle, the second height can be set based on the upper limit value of the height of the vehicle that can travel on the road. The height of the vehicle that can travel on the road is defined by traffic regulations and the like. For example, in the Road Traffic Law of Japan, the height of a truck is, in principle, 3.8 m or less. In this case, the second height can be 4 m. By removing the parallax corresponding to an object with a height from the road surface of the second height or more from the first parallax image, the information on the parallax corresponding to the object is removed from the first parallax image. In Figure 18 In the second parallax image 60 shown, by removing the parallax corresponding to an object with a height from the road surface of the second height or more, the parallax image corresponding to the object located on the negative direction side of the v axis does not contain parallax information. By removing the information on the parallax corresponding to an object of the second height or more, in subsequent processing, for the Figure 18 parallax information of the parallax image 42 corresponding to another vehicle can be processed more easily. Further, by removing unnecessary parallax that is not related to the object to be detected, the processing speed described later can be increased.

[0157] After the processing of step S103 is executed, the control unit 24 detects the first parallax and the second parallax from the second parallax image (step S104). In Figure 20 The detailed content of the processing of step S104 is shown in the flowchart shown.

[0158] In the processing of step S501, the control unit 24 divides the second parallax image into a plurality of partial regions by dividing it by Δu1 along the u direction. In Figure 21In [description], a partial region 61 is represented overlapping the second parallax image 60. The partial region 61 may be a rectangle whose long side is significantly longer than the short side. The long side of the partial region 61 is more than 5 times the short side of the partial region 61. The long side of the partial region 61 may be more than 10 times the short side of the partial region 61, or may be more than 20 times the short side of the partial region 61. The short side of the partial region 61, i.e., Δu1, may be several pixels to several dozen pixels. As described later, based on the partial region 61, the first parallax and the second parallax can be detected. The shorter the short side of the partial region 61, i.e., Δu1, the higher the resolution of the detection of the first parallax and the second parallax. The control unit 24 can sequentially acquire the partial region 61 from the negative direction side to the positive direction side of the u-axis shown in Figure 21 and perform the processing of steps S502 to S506 below.

[0159] In the processing of step S502, the control unit 24 generates a parallax histogram for each partial region. An example of the parallax histogram is shown in Figure 22 . Figure 22 The horizontal axis of [description] corresponds to the magnitude of the parallax. Figure 22 The vertical axis of [description] corresponds to the frequency of the parallax. The frequency of the parallax represents the number of parallax pixels of the parallax included in the partial region. As shown in Figure 21 , in the partial region 61, multiple parallax pixels in the parallax image corresponding to the same object may be included. For example, multiple parallax pixels 42a are included in the partial region 61-1. The multiple parallax pixels 42a are the parallax pixels included in the partial region 61-1 among the multiple parallax pixels in the parallax image 42 corresponding to another vehicle. In the partial region 61, the parallaxes represented by the multiple parallax pixels corresponding to the same object may be of the same degree. For example, in the partial region 61-1, the parallaxes represented by the multiple parallax pixels 42a may be of the same degree. That is, in the parallax histogram shown in Figure 22 , the frequency of the parallax corresponding to the same object can be increased.

[0160] In the processing of step S502, the control unit 24 can Figure 22The width of the interval Sn of the parallax histogram shown expands as the parallax becomes smaller. The interval Sn is the n-th interval of the parallax histogram counted from the side with a smaller parallax. The starting point of the interval Sn is the parallax dn-1. The ending point of the interval Sn is the parallax dn. For example, the control unit 24 can make the width of the interval Sn-1 wider than the width of the interval Sn by about 10% of the width of the interval Sn. When the distance from the stereo camera 10 to the object is far, compared with the case where the distance from the stereo camera 10 to the object is near, the parallax corresponding to the object may become smaller. When the distance from the stereo camera 10 to the object is far, compared with the case where the distance from the stereo camera 10 to the object is near, the number of pixels occupied by the object in the stereo image may become smaller. That is, when the distance from the stereo camera 10 to the object is far, compared with the case where the distance from the stereo camera 10 to the object is near, the number of parallax pixels representing the parallax corresponding to the object in the second parallax image can become smaller. As the parallax becomes smaller, Figure 22 the width of the interval Sn of the parallax histogram shown becomes wider, so that the parallax corresponding to an object at a distance far from the stereo camera 10 can be easily detected by the processing in steps S503 to S506 described later. The control unit 24 can sequentially execute the processing in steps S503 to S506 for each interval of the parallax histogram.

[0161] In the processing of step S503, the control unit 24 determines whether there is an interval Sn in the generated parallax histogram in which the frequency of the parallax exceeds a first threshold Tr1 (prescribed threshold). When the control unit 24 determines that there is an interval Sn in which the frequency of the parallax exceeds the first threshold Tr1 (step S503: Yes), it detects the parallax in the range from the starting point (parallax dn-1) to the ending point (parallax dn) of the interval Sn as the first parallax (step S504). The control unit 24 regards the first parallax as the parallax corresponding to the object and detects it. The control unit 24 stores the detected first parallax in association with the u coordinate in the memory 23. On the other hand, when the control unit 24 determines that there is no interval Sn in the generated parallax histogram in which the frequency of the parallax exceeds the first threshold Tr1 (step S503: No), it proceeds to the processing of step S505.

[0162] The first threshold Tr1 can be set based on the minimum value of the height of an object that is the detection target of the object detection device 20 (for example, the height of a child is 50 cm). However, as described above, when the object is at a distance of 10 from the stereo camera, the parallax corresponding to the object can be smaller than when the object is at a distance of 10 close to the stereo camera. In this case, if the first threshold Tr1 is kept constant with respect to the parallax, there is a case where it is difficult to detect the parallax corresponding to an object at a distance of 10 from the stereo camera compared to an object at a distance of 10 close to the stereo camera. Therefore, based on the minimum value of the height of an object that is the detection target of the object detection device 20, the first threshold Tr1 can be set to increase as the parallax in the parallax histogram becomes smaller. For example, the first threshold Tr1 can be calculated by the mathematical formula (2).

[0163] Tr1 = (D × H) / B (2)

[0164] In the mathematical formula (2), the parallax D is the parallax corresponding to the horizontal axis of the parallax histogram. The height H is the minimum value of the height of an object that is the detection target of the object detection device 20. The baseline length B is the distance between the optical center of the first camera 11 and the optical center of the second camera 12 (baseline length).

[0165] The process of step S505 is a process of determining whether to detect the second parallax. Before explaining the process of step S505, the reason for detecting the second parallax will be explained. As described above, in the matching process for calculating the parallax, since there are few feature amounts on the stereo image, the parallax of a part of the parallax image corresponding to the object may not be calculated or may be smaller than the parallax of other parts. For example, as Figure 19 shown, since there are few feature amounts on the stereo image, the parallax of the central part on the lower side of the parallax image 44 corresponding to another vehicle cannot be calculated. If a part of the parallax of the parallax image corresponding to the object is not calculated, then for example, even if an object with an actual height of 50 cm or more in the actual space can be displayed as two or more independent objects with an actual height less than 50 cm (for example, 10 cm) in the second parallax image in the actual space. That is, even if there is an object with an actual height in the actual space above the height of the detection target, if a part of the parallax image corresponding to the object is missing, through the determination process of the above first threshold Tr1, the parallax corresponding to the object may not be detected as the first parallax. Therefore, in the present embodiment, such a parallax that is not detected as the first parallax is detected as a candidate for the parallax corresponding to the object, that is, the second parallax. By detecting it as the second parallax, it is possible to determine whether to restore the parallax of the object to be detected in the process of step S107 described later as described later Figure 5 shown in the later description.

[0166] In the process of step S505, the control unit 24 determines whether there is an interval Sn in the generated disparity histogram where the frequency of disparity is below the first threshold Tr1 and exceeds the second threshold Tr2 (prescribed threshold).

[0167] The second threshold Tr2 can be a prescribed ratio of the first threshold Tr1. In the disparity image of the same object, the prescribed ratio is appropriately set based on the ratio between the part where the disparity is not calculated and the part where the disparity is calculated. For example, the prescribed ratio can be 0.2.

[0168] In the process of step S505, when the control unit 24 determines that there is an interval Sn in the generated disparity histogram where the frequency of disparity is below the first threshold Tr1 and exceeds the second threshold Tr2 (step S505: Yes), it proceeds to the process of step S506. On the other hand, when the control unit 24 determines that there is no interval Sn in the generated disparity histogram where the frequency of disparity is below the first threshold Tr1 and exceeds the second threshold Tr2 (step S505: No), it returns to Figure 5 the process of step S105 shown.

[0169] In the process of step S506, the control unit 24 detects the disparity in the range from the start point (disparity dn-1) to the end point (disparity dn) of this interval Sn as the second disparity (step S506). The control unit 24 stores the detected second disparity in association with the u coordinate in the memory 23. After executing the process of step S506, the control unit 24 returns to Figure 5 the process of step S105 shown.

[0170] In the process of step S105, the control unit 24 calculates the height of the object on the image. In Figures 23 to 25 the detailed content of the process of step S105 is shown in the flowchart.

[0171] Hereinafter, the control unit 24 performs a calculation process for calculating the height of the object on the second disparity map, which is the second disparity image. However, the control unit 24 can perform a calculation process for calculating the height of the object on any disparity map in which the disparity is associated with the uv coordinates. In the calculation process of the height of the object, the control unit 24 calculates the height of the object on the image by scanning the disparity pixels in the v direction among the u coordinates of the search target. For example, the control unit 24 calculates the height of the object on the image by scanning the disparity pixels representing the first disparity regarded as the disparity corresponding to the object and / or the disparity image representing the second disparity as a candidate for the disparity corresponding to the object in the v direction.

[0172] When starting the processing of step S601, the control unit 24 acquires the minimum coordinate of the u coordinate as the u coordinate of the search target. In the present embodiment, when used for scanning on an image, the minimum coordinate and the maximum coordinate refer to the minimum coordinate and the maximum coordinate of the scanning range. The minimum coordinate and the maximum coordinate of the scanning range may not coincide with the minimum coordinate and the maximum coordinate on the image. The minimum coordinate and the maximum coordinate of the scanning range can be arbitrarily set.

[0173] In the processing of step S601, the control unit 24 refers to the memory 23 and determines whether there is a first parallax or a second parallax detected through the process shown in Figure 20 that corresponds to the u coordinate of the search target.

[0174] In Figure 26 the second parallax image 70 is shown. In Figure 26 a part of the second parallax image 70 is shown. The second parallax image 70 corresponds to the region 62 of the second parallax image 60 shown in Figure 18 The second parallax image 70 includes a parallax image 42 of another vehicle. In the second parallax image 70, the shaded part is a parallax pixel representing the first parallax or the second parallax. When the u coordinate of the search target is the coordinate (u0), the control unit 24 determines that there is no first parallax and second parallax corresponding to the u coordinate of the search target. When the u coordinate of the search target is the coordinate (u1), the control unit 24 determines that there is a first parallax or a second parallax corresponding to the u coordinate of the search target.

[0175] In the processing of step S601, when the control unit 24 does not determine that there is a first parallax or a second parallax corresponding to the u coordinate of the search target (step S601: No), it proceeds to the processing of step S602. On the other hand, when the control unit 24 determines that there is a first parallax or a second parallax corresponding to the u coordinate of the search target (step S601: Yes), it proceeds to the processing of step S604.

[0176] In the processing of step S602, the control unit 24 determines whether the u coordinate of the search target is the maximum coordinate. When the u coordinate of the search target is the maximum coordinate, it means that the control unit 24 has completely searched the u coordinate from the minimum coordinate to the maximum coordinate. When the control unit 24 determines that the u coordinate of the search target is the maximum coordinate (step S602: Yes), it ends Figure 23 the processing shown in Figure 5 and returns to the processing of step S106 shown in Figure 26When at the coordinate (u0) shown, the control unit 24 determines that the u coordinate of the search target is not the maximum coordinate. On the other hand, when the control unit 24 determines that the u coordinate of the search target is not the maximum coordinate (step S602: No), it increments the u coordinate of the search target by 1 (step S603). The control unit 24 performs the process of step S601 on the u coordinate obtained by incrementing by 1 through the process of step S603. For example, the control unit 24 can start from the coordinate (u0) shown in Figure 26 and repeat the processes of steps S601 to S603 on the u coordinate of the search target until it is determined that there is a first parallax or a second parallax corresponding to the u coordinate of the search target.

[0177] In the process of step S604, the control unit 24 acquires the object parallax. The object parallax can be only the first parallax detected as the parallax regarded as corresponding to the object. In other words, the object parallax can be the parallax that satisfies the determination process of step S503 shown in Figure 20 . Or, the object parallax can include both the first parallax and the second parallax. In other words, the object parallax can be the parallax that satisfies the determination process of step S503 and the determination process of step S505 shown in Figure 20 . Hereinafter, in the process shown in Figures 23 to 25 , the case where the object parallax includes both the first parallax and the second parallax will be described.

[0178] In the process of step S604, the control unit 24 acquires the maximum parallax as the object parallax among the first parallax and the second parallax corresponding to the u coordinate of the search target stored in the memory 23.

[0179] In the process of step S605, the control unit 24 calculates the v coordinate of the road surface corresponding to the object parallax. For example, the control unit 24 substitutes the object parallax into the road surface parallax d in the approximate formula ( Figure 7 ) representing the relationship between the v coordinate of the road surface obtained in the process of step S211 shown in r and the road surface parallax d Figure 17 ) to calculate the v coordinate of the road surface. In r , it is assumed that the u coordinate of the search target is the coordinate (u1). In this case, the control unit 24 calculates the coordinate (v1) as the v coordinate of the road surface. Figure 26

[0180] ​In the process of step S606, the control unit 24 determines whether there are coordinates corresponding to a parallax approximately equal to the object parallax within a specified range along the negative direction of the v-axis based on the calculated v coordinate of the road surface. In the present invention, the "parallax approximately equal to the object parallax" includes: the same parallax as the object parallax and a parallax substantially the same as the object parallax. In the present invention, the "parallax substantially the same as the object parallax" refers to a parallax that can be regarded as the object parallax in image processing. For example, a parallax with a difference from the object parallax within the range of ±10% can be regarded as a parallax approximately equal to the object parallax. In addition, the specified range in the process of step S606 can be appropriately set based on the height of the object floating from the road surface. As an example of the object floating from the road surface, street trees, pedestrian bridges, traffic lights, etc. can be cited. Such an object floating from the road surface is an object other than the detection object of the object detection device 20. That is, for such an object floating from the road surface, the height does not need to be calculated.

[0181] In the process of step S606, when the control unit 24 does not determine that there are coordinates corresponding to a parallax approximately equal to the object parallax within the specified range along the negative direction of the v-axis starting from the v coordinate of the road surface (step S606: No), it proceeds to the process of step S607 where the height of the object is not calculated. In the process of step S607, the control unit 24 adds a scanned flag to the object parallax obtained in the process of step S604 and stored in the memory 23. After the control unit 24 executes the process of step S607, it executes the process of step S601 again. In the process of step S601 executed again, the control unit 24 determines whether there is a first parallax or a second parallax without the scanned flag for the u coordinate of the search object. In the process of step S604 executed again, among the first parallax and the second parallax without the scanned flag, the maximum parallax is obtained as the object parallax.

[0182] In the process of step S606, when the control unit 24 determines that there are coordinates corresponding to a parallax approximately equal to the object parallax within the specified range along the negative direction of the v-axis starting from the v coordinate of the road surface (step S606: Yes), it proceeds to the process of step S608. In the process of step S608, the control unit 24 obtains the coordinates corresponding to the parallax approximately equal to the object parallax existing within the specified range along the negative direction of the v-axis starting from the v coordinate of the road surface as the first coordinates. In Figure 26In , it is assumed that the u coordinate of the search object is the coordinate (u1). Additionally, it is assumed that a parallax approximately equal to the object parallax is associated with the coordinate (u1, v2) of the parallax pixel 71. That is, it is assumed that the parallax pixel 71 represents a parallax approximately equal to the object parallax. Further, it is assumed that the coordinate (v0) of the v coordinate of the road surface and the coordinate (v1) of the v coordinate of the parallax pixel 71 are within a specified range. In this case, the control unit 24 acquires the coordinate (u1, v2) of the parallax pixel 71 as the first coordinate.

[0183] After performing the process of step S608, the control unit 24 determines whether a parallax approximately equal to the object parallax is associated with the coordinate obtained by decreasing the v coordinate of the first coordinate by 1 (step S609). In Figure 26 In , it is assumed that the first coordinate is the coordinate (u1, v2) of the parallax pixel 71. The coordinate obtained by decreasing the v coordinate of the parallax pixel 71 by 1 is the coordinate (u1, v3) of the parallax pixel 72. It is assumed that the parallax pixel 72 represents a parallax approximately equal to the object parallax. That is, it is assumed that a parallax approximately equal to the object parallax is associated with the coordinate (u1, v3) of the parallax pixel 72. In this case, the control unit 24 determines that a parallax approximately equal to the object parallax is associated with the coordinate (u1, v3) of the parallax pixel 72 obtained by decreasing the v coordinate of the coordinate (u1, v2) as the first coordinate by 1.

[0184] In the process of step S609, when the control unit 24 determines that a parallax approximately equal to the object parallax is associated with the coordinate obtained by decreasing the v coordinate of the first coordinate by 1 (step S609: YES), it proceeds to the process of step S610. In the process of step S610, the control unit 24 updates the coordinate obtained by decreasing the v coordinate of the first coordinate by 1 as the first coordinate. In Figure 26 In , when the first coordinate is the coordinate (u1, v2) of the parallax pixel 71, through the process of step S610, the first coordinate is updated to the coordinate (u1, v3) of the parallax pixel 72. After performing the process of step S610, the control unit 24 returns to the process of step S609. For example, the control unit 24 repeatedly executes the process of step S609 and the process of step S610 until Figure 26 the coordinate (u1, v4) of the parallax pixel 73 shown in is updated as the first coordinate. A parallax approximately equal to the object parallax is associated with the coordinate (u1, v4) of the parallax pixel 73.

[0185] In the process of step S609, when the control unit 24 determines that a parallax approximately equal to the object parallax is not associated with the coordinate obtained by decreasing the v coordinate of the first coordinate by 1 (step S609: NO), it proceeds to Figure 24 the process of step S611 shown in . In Figure 26In this case, it is assumed that the first coordinate is the coordinate (u1, v4) of the parallax pixel 73. The coordinate obtained by decreasing the v coordinate of the parallax pixel 73 by 1 is the coordinate (u1, v5) of the parallax pixel 74. The parallax pixel 74 represents a parallax smaller than the parallax approximately equal to the object parallax. That is, the parallax approximately equal to the object parallax does not correspond to the coordinate (u1, v5) of the parallax pixel 74. In this case, the control unit 24 determines that the parallax approximately equal to the object parallax does not correspond to the coordinate (u1, v5) of the parallax pixel 74 obtained by decreasing the v coordinate of the coordinate (u1, v4) as the first coordinate by 1.

[0186] In the process of step S611, the control unit 24 determines whether the v coordinate of the first coordinate is the minimum coordinate. When the v coordinate of the first coordinate is the minimum coordinate, it means that the control unit 24 has completely scanned the second parallax image along the negative direction of the v axis. When the control unit 24 determines that the v coordinate of the first coordinate is the minimum coordinate (step S611: Yes), it proceeds to the process of step S616. On the other hand, when the control unit 24 determines that the v coordinate of the first coordinate is not the minimum coordinate (step S611: No), it proceeds to the process of step S612. For example, when the first coordinate is Figure 26 the coordinate (u1, v4) of the parallax pixel 73 shown, the control unit 24 determines that the v coordinate of the first coordinate (coordinate (v4)) is not the minimum coordinate.

[0187] Before describing the process of step S612, please refer to Figure 26 . In Figure 26 this case, it is assumed that the first coordinate is the coordinate (u1, v4) of the parallax pixel 73. There is a parallax pixel 75 on the negative side of the v axis of the parallax pixel 73. It is assumed that the parallax pixel 75 represents a parallax approximately equal to the object parallax. The parallax pixel 73 and the parallax pixel 75 are part of the parallax image 42 corresponding to other vehicles as the same object. However, due to parallax deviation, etc., even parallax pixels included in the parallax image corresponding to the same object may be arranged separately in the v direction like the parallax pixel 73 and the parallax pixel 75.

[0188] Therefore, in the process of step S612, the control unit 24 determines whether there is a coordinate corresponding to the parallax approximately equal to the object parallax within a specified interval along the negative direction of the v axis starting from the v coordinate of the first coordinate. The specified interval can be appropriately set based on the height of the object to be detected by the object detection device 20. For example, the specified interval can be appropriately set based on the height of the rear of the vehicle (e.g., 80 cm). In Figure 26Among them, the interval in the v direction between the parallax pixel 73 and the parallax pixel 75 is within a specified interval. When the first coordinate is the coordinate (u1, v4) of the parallax pixel 73, the control unit 24 determines that there exists a coordinate (u1, v6) of the parallax pixel 75 corresponding to a parallax approximately equal to the object parallax.

[0189] In the process of step S612, when the control unit 24 determines that there exists a coordinate corresponding to a parallax approximately equal to the object parallax within a specified interval in the negative direction of the v-axis starting from the v coordinate of the first coordinate (step S612: Yes), it proceeds to the process of step S613. On the other hand, when the control unit 24 determines that there does not exist a coordinate corresponding to a parallax approximately equal to the object parallax within a specified interval in the negative direction of the v-axis starting from the v coordinate of the first coordinate (step S612: No), it proceeds to the process of step S615.

[0190] The purpose of the process of step S613 is to determine whether the object parallax and the parallax approximately equal to the object parallax in the process of step S612 are parallaxes corresponding to the same object. Before explaining the process of step S613, refer to Figure 27 and Figure 28 , and explain an example of the second parallax image.

[0191] In Figure 27 shows the second parallax image 80. The second parallax image 80 is an image generated based on Figure 28 the first image 90 shown. The first image 90 includes: an image 91 corresponding to another vehicle, an image 92 corresponding to a street tree, and an image 93 corresponding to a building. As Figure 27 shown, the second parallax image 80 includes: a parallax image 81 corresponding to Figure 28 the image 91 shown, a parallax image 82 corresponding to Figure 28 the image 92 shown, and a parallax image 85 corresponding to Figure 28 the image 93 shown.

[0192] As Figure 27 shown, the parallax image 81 includes a parallax pixel 83. The parallax image 82 includes a parallax pixel 84. The u coordinates of the parallax pixel 83 and the parallax pixel 84 are the same at the coordinate (u2). In addition, in the real space, the distance from the stereo camera 10 to the other vehicle corresponding to Figure 28 the image 91 shown is approximately equal to the distance from the stereo camera 10 to the street tree corresponding to Figure 28 the image 92 shown. Since these two distances are approximately equal, Figure 27 the parallax represented by the parallax pixel 83 of the parallax image 81 shown in Figure 27The disparities represented by the disparity pixels 84 of the disparity image 82 shown may be approximately equal. Additionally, in the height direction of the actual space, the distance between other vehicles corresponding to the Figure 28 image 92 shown and street trees corresponding to the Figure 28 image 92 shown is relatively close. Since these two distances are close, the interval between the disparity pixel 83 of the disparity image 81 and the disparity pixel 84 of the disparity image 82 in the Figure 27 v direction shown is within the above-mentioned specified interval in the process of step S612.

[0193] In the Figure 27 structure shown, when the u coordinate of the search object is set to the coordinate (u2), the control unit 24 acquires the disparities represented by the disparity pixel 83 and the disparity pixel 84 as the object disparity. Additionally, the control unit 24 can update the coordinate of the disparity pixel 83 to the first coordinate. When the u coordinate of the search object is the coordinate (u2), it is preferable to calculate the height T1 corresponding to the disparity image 81 as the height of the object on the image. As an exemplary scenario, when the first coordinate is the coordinate of the disparity pixel 83, it is considered that since the interval between the disparity pixel 83 and the disparity pixel 84 is within the above-mentioned specified interval, the control unit 24 updates the coordinate of the disparity pixel 84 to the first coordinate. In this exemplary scenario, the control unit 24 can continuously scan the disparity pixels of the disparity image 82 corresponding to the street trees. As a result, the control unit 24 detects the height T2 relative to the disparity image 82, that is, the height corresponding to the street trees, as the height of the object.

[0194] Among them, there are multiple disparity pixels 86 between the disparity pixel 83 and the disparity pixel 84. The multiple disparity pixels 86 are included in the disparity image 85. The building corresponding to the disparity image 85 is located at a position farther from the stereo camera 10 than the vehicle and the street trees. Therefore, the disparities represented by the multiple disparity pixels 86 of the disparity image 85 are smaller than the disparities represented by the disparity pixel 83 and the disparity pixel 84.

[0195] Therefore, in the process of step S613, the control unit 24 determines whether there are coordinates corresponding to a third disparity exceeding a specified number between the first coordinate and the second coordinate. The second coordinate is a coordinate within a specified interval along the negative direction of the v axis from the first coordinate and is a coordinate corresponding to a disparity approximately equal to the object disparity. For example, when the first coordinate is the Figure 27 coordinate of the disparity pixel 83 shown, the second coordinate becomes the coordinate of the disparity pixel 84. The third disparity is a disparity smaller than the object disparity. The third disparity can be set based on the disparity corresponding to the background. For example, it can be assumed that by Figure 27Set the third parallax based on the parallax represented by the parallax pixel 86 shown. In the above-mentioned specified interval, the specified number can be appropriately set by assuming the number of parallax pixels including the parallax pixel representing the third parallax.

[0196] In the process of step S613, when the control unit 24 determines that there are more than the specified number of coordinates corresponding to the third parallax between the first coordinate and the second coordinate (step S613: Yes), it proceeds to the process of step S616. For example, when the first coordinate and the second coordinate are Figure 27 the coordinates of the parallax pixels 83 and 84 shown, the control unit 24 determines that there are more than the specified number of parallax pixels 86 between the coordinates of the parallax pixel 83 and the coordinates of the parallax pixel 84. On the other hand, when the control unit 24 determines that there are no more than the specified number of coordinates corresponding to the third parallax between the first coordinate and the second coordinate (step S613: No), it proceeds to the process of step S614. For example, when the first coordinate and the second coordinate are Figure 26 the coordinates of the parallax pixel 73 and the coordinates of the parallax pixel 75 shown, the control unit 24 determines that there are no more than the specified number of parallax pixels 74 between the coordinates of the parallax pixel 73 and the coordinates of the parallax pixel 75.

[0197] In the process of step S614, the control unit 24 updates the second coordinate to the first coordinate. For example, when the first coordinate and the second coordinate are Figure 26 the coordinates of the parallax pixel 73 and the coordinates of the parallax pixel 75 shown, the control unit 24 updates the coordinates (u1, v6) of the parallax pixel 75 to the first coordinate. After performing the process of step S614, the control unit 24 returns to Figure 23 the process of step S609 shown. For example, in Figure 26 the structure shown, the control unit 24 can repeatedly execute the process starting from step S609 until the coordinates (u1, v7) of the parallax pixel 76 are updated to the first coordinate position.

[0198] In the process of step S615, the control unit 24 determines whether there are coordinates corresponding to a parallax approximately equal to the object parallax beyond the specified interval along the negative direction of the v-axis starting from the first coordinate. For example, when the first coordinate is Figure 26When the coordinates (u1, v7) of the parallax pixel 76 shown are obtained, the control unit 24 determines that there are no coordinates corresponding to a parallax approximately equal to the object parallax beyond a specified interval along the negative direction of the v-axis starting from the coordinates (u1, v7). When the control unit 24 determines that there are no coordinates corresponding to a parallax approximately equal to the object parallax beyond a specified interval along the negative direction of the v-axis starting from the first coordinates (step S615: No), it proceeds to the process of step S616. On the other hand, when the control unit 24 determines that there are coordinates corresponding to a parallax approximately equal to the object parallax beyond a specified interval along the negative direction of the v-axis starting from the first coordinates (step S615: Yes), it proceeds to Figure 25 the process of step S618 shown.

[0199] In the process of step S616, the control unit 24 calculates the height of the object on the image by subtracting the v-coordinate of the road surface coordinates from the v-coordinate of the first coordinates. When the first coordinates are Figure 26 the coordinates (u1, v7) of the parallax pixel 76 shown, the control unit 24 calculates the height of the object by subtracting the v-coordinate of the road surface coordinates (coordinate (v1)) from the v-coordinate of the parallax pixel 76 (coordinate (v7)). When the first coordinates are Figure 27 the coordinates of the parallax pixel 83 shown, the control unit 24 calculates the height T1 of the object by subtracting the v-coordinate of the road surface (coordinate (v8)) from the v-coordinate of the parallax pixel 83. As Figure 27 shown, through the processes of steps S613 and S616, even if the parallax image 81 of another vehicle and the parallax image 82 of a street tree are relatively close, the height of the other vehicle can be calculated with high precision. In the process of step S616, the control unit 24 stores the calculated height of the object on the image in association with the first coordinates in the memory 23.

[0200] In the process of step S616, the control unit 24 may transform the calculated height of the object on the image into the height of the object in the actual space. In the case where the height of the object in the actual space is less than the minimum value of the height of the object to be detected, the control unit 24 may discard the information on the calculated height of the object. For example, when the minimum value of the height of the object to be detected is 50 cm, the control unit 24 may discard the information on the calculated height of the object when the height of the object in the actual space is less than 40 cm.

[0201] After executing the process of step S616, the control unit 24 executes the process of step S617. In the process of step S617, the control unit 24 attaches a scanned mark to the object parallax. After executing the process of step S617, the control unit 24 returns to Figure 23 the process of step S601 shown.

[0202] Before describing the process of step S618, refer to Figure 29 and describe an example of the second parallax image. In Figure 29 , the second parallax image 100 is shown. In Figure 29 , a part of the second parallax image 100 is shown. The parallax image 101 is a parallax image generated based on Figure 30 the first image 106 shown. The first image 106 includes an image 107 corresponding to the rear of the truck.

[0203] As Figure 29 shown, the second parallax image 100 contains the parallax image 101. The parallax image 101 corresponds to Figure 30 the image 107 shown. The parallax image 101 includes parallax pixels 102, parallax pixels 103, parallax pixels 104, and parallax pixels 105. The parallax pixel 103 corresponds to Figure 30 the lower image 107a of the image 107 shown. The parallax pixels 104 and 105 correspond to Figure 30 the upper image 107b of the image 107 shown. Since the parallax represented by the parallax pixel 103, the parallax represented by the parallax pixel 104, and the parallax represented by the parallax pixel 105 are parallaxes corresponding to the same object, the truck, they are substantially equal.

[0204] As Figure 29 shown, a plurality of parallax pixels 102 are located in the central part of the parallax image 101. Among the plurality of parallax pixels 102, the parallax is not calculated. That is, the plurality of parallax pixels 102 do not include parallax information. Generally, in the central part of the rear of the truck, the feature amount on the stereo image is small. In the central part of the rear of the truck, since the feature amount on the stereo image is small, in the above-described matching process for calculating the parallax, sometimes the parallax is not calculated like the parallax pixel 102.

[0205] In Figure 29 the structure shown, as a hypothetical example, consider an example in which the control unit 24 scans the u coordinate of the search target along the v direction as the coordinate (u3). In the hypothetical example, it is preferable to calculate the height T3 corresponding to the parallax image 101, that is, the height of the truck, as the height of the object. In the hypothetical example, the control unit 24 can obtain the parallax represented by the parallax pixel 103 as the target parallax. In addition, the control unit 24 updates the coordinate (u3, v 10 ) of the parallax pixel 103 to the first coordinate. In order to calculate the height T3, in the process of step S614 described above, it is necessary to update the coordinate (u3, v 11 ) of the parallax pixel 104 to the first coordinate. In order to update the coordinate (u3, v 11)Updated to the first coordinate, it is necessary to make the specified interval in step S612 above wider than the interval between the v coordinate (coordinate (v 11 )) of the parallax pixel 104 and the v coordinate (coordinate (v 10 )) of the parallax pixel 103. However, if the specified interval in the process of step S612 above is widened, the possibility of miscomputing the height of the object becomes higher.

[0206] Therefore, the control unit 24 calculates by executing the processes of steps S618 to S622 Figure 29 The height T4 shown is the first candidate height, and calculates Figure 29 The height T3 shown is the second candidate height. In step S107 described later Figure 5 shown, the control unit 24 determines which of the first candidate height and the second candidate height to obtain as the height of the object. With such a structure, the specified interval in the process of step S612 above can be not widened. Since the specified interval in the process of step S612 above is not widened, the possibility of miscomputing the height of the object can be reduced.

[0207] In the process of step S618, the control unit 24 obtains candidate coordinates. The candidate coordinates are coordinates that exist beyond the specified interval in the negative direction of the v-axis starting from the first coordinate, and are coordinates corresponding to a parallax approximately equal to the target parallax. For example, when the first coordinate is Figure 29 the coordinates (u3, v 10 ) of the parallax pixel 103 shown, the control unit 24 obtains the coordinates (u3, v 11 ) of the parallax pixel 104 as candidate coordinates.

[0208] In the process of step S619, the control unit 24 calculates the first candidate height by subtracting the v coordinate of the road surface coordinates from the v coordinate of the first coordinate. For example, when the first coordinate is Figure 29 the coordinates (u3, v 10 ) of the parallax pixel 103 shown, the control unit 24 calculates the first candidate height T4 by subtracting the v coordinate (coordinate (v 10 )) of the parallax pixel 103 from the v coordinate (coordinate (v9)) of the road surface. In the process of step S619, the control unit 24 stores the calculated first candidate height in association with the first coordinate in the memory 23. For example, the control unit 24 Figure 29 stores the first candidate height T4 shown in association with the coordinates (u3, v 10 ) of the parallax pixel 103 of the first coordinate in the memory 23.

[0209] In the process of step S620, the control unit 24 determines whether the parallax approximately equal to the object parallax corresponds to the coordinates obtained by reducing the v coordinate of the candidate coordinates by 1. When the control unit 24 determines that the parallax approximately equal to the object parallax corresponds to the coordinates obtained by reducing the v coordinate of the candidate coordinates by 1 (step S620: Yes), it proceeds to the process of step S621. In the process of step S621, the control unit 24 updates the coordinates obtained by reducing the v coordinate of the candidate coordinates by 1 as the candidate coordinates. After executing the process of step S621, the control unit 24 returns to the process of step S620. For example, in Figure 29 the structure shown, the control unit 24 repeatedly executes the processes of step S620 and step S621 until the coordinates (u3, v 12 ) of the parallax pixel 105 are updated as the candidate coordinates. On the other hand, when the control unit 24 determines that the parallax approximately equal to the object parallax does not correspond to the coordinates obtained by reducing the v coordinate of the candidate coordinates by 1 (step S620: No), it proceeds to the process of step S622. For example, when the candidate coordinates are Figure 29 the coordinates (u3, v 12 ) of the parallax pixel 105 shown, the control unit 24 determines that the parallax approximately equal to the object parallax does not correspond to the coordinates obtained by reducing the v coordinate of the candidate coordinates by 1.

[0210] In the process of step S622, the control unit 24 calculates the second candidate height by subtracting the v coordinate of the road surface from the v coordinate of the candidate coordinates. For example, when the candidate coordinates are Figure 29 the coordinates (u3, v 12 ) of the parallax pixel 105 shown, the control unit 24 calculates the second candidate height T3 by subtracting the v coordinate of the road surface (coordinate (v9)) from the v coordinate of the parallax pixel 105 (coordinate (v 12 )). In the process of step S622, the control unit 24 stores the calculated second candidate height in the memory 23 in association with the candidate coordinates. For example, the control unit 24 stores Figure 29 the second candidate height T3 shown in association with the coordinates (u3, v 12 ) of the parallax pixel 105 of the candidate coordinates in the memory 23. After executing the process of step S622, the control unit 24 proceeds to the process of step S623.

[0211] In the process of step S623, the control unit 24 attaches a scanned mark to the object parallax obtained in the process of step S604 and stored in the memory 23. After executing the process of step S623, the control unit 24 returns to Figure 23 the process of step S601 shown.

[0212] In the process of step S106, the control unit 24 performs a detection process for parallel objects. The detailed content of the process of step S106 is shown in the flowchart shown in Figure 31 .

[0213] In the process of step S701, the control unit 24 generates or acquires a UD map. The UD map is also referred to as the "U-disparity space" and the "u-d coordinate space". The UD map is a map in which a two-dimensional coordinate composed of the u direction and the d direction corresponding to the magnitude of the disparity is associated with the object disparity. The object disparity may be only the first disparity detected as the disparity corresponding to the object. Or, the object disparity may include both the first disparity and the second disparity. Hereinafter, it is assumed that the object disparity includes the first disparity and the second disparity. The control unit 24 may acquire the first disparity and the second disparity stored in the memory 23 and generate a UD map. Or, the control unit 24 may acquire the UD map from the outside through the acquisition unit 21.

[0214] The UD map 110 is shown in Figure 32 . The horizontal axis of the UD map 110 corresponds to the u axis. The vertical axis of the UD map 110 corresponds to the d axis representing the magnitude of the disparity. The coordinate system composed of the u coordinate and the d coordinate is also referred to as the "ud coordinate system". In the UD map 110, the corner facing the lower left side of the paper surface is taken as the origin (0, 0) of the ud coordinate system. Figure 32 Figure 32 The plotted points shown are the coordinate points corresponding to the first disparity and the second disparity detected by the process shown in Figure 20 . The coordinate points are also simply referred to as "points". The UD map 110 includes a point group 111, a point group 112, a point group 113, a point group 114, and a point group 115. The UD map 110 is a map generated based on the second disparity image generated from the first image 120 shown in Figure 33 .

[0215] As shown in Figure 33 , the first image 120 includes: an image 121 corresponding to a guardrail, images 122 and 123 corresponding to other vehicles, an image 124 corresponding to a pedestrian, and an image 125 corresponding to a side wall. The first disparity and the second disparity detected from each of the images 121 to 125 correspond to each of the point groups 111 to 115 shown in Figure 32 . The images 121 and 125 may be images corresponding to parallel objects. That is, the point groups 111 and 115 shown in Figure 32 may be disparities corresponding to parallel objects. One of the purposes of the detection process for parallel objects is to detect disparities corresponding to parallel objects such as the point groups 111 and 115 shown in Figure 32 .

[0216] In the present embodiment, by applying the Hough transform in the process of step S704 described later, the parallax corresponding to parallel objects such as the point group 111 and the point group 115 shown in the figure is detected. However, as shown in Figure 32 , the point group 112 and the point group 113 are located near the point group 111 and the point group 115. If the point group 112 and the point group 113 are located near the point group 111 and the point group 115, the accuracy of detecting the point group 111 and the point group 115 can be reduced by the Hough transform in the process of step S704 described later. Therefore, through the process of step S702, the control unit 24 determines whether there are point groups 112 and 113 such as those shown in Figure 32 that are located near the point group 111 and the point group 115. Figure 32 As shown, the parallax corresponding to parallel objects such as the point group 111 and the point group 115 is detected. However, as shown in Figure 32 , the point group 112 and the point group 113 are located near the point group 111 and the point group 115. If the point group 112 and the point group 113 are located near the point group 111 and the point group 115, the accuracy of detecting the point group 111 and the point group 115 can be reduced by the Hough transform in the process of step S704 described later. Therefore, through the process of step S702, the control unit 24 determines whether there are point groups 112 and 113 such as those shown in Figure 32 that are located near the point group 111 and the point group 115. Figure 32 As shown, the point group 112 and the point group 113 are located near the point group 111 and the point group 115. If the point group 112 and the point group 113 are located near the point group 111 and the point group 115, the accuracy of detecting the point group 111 and the point group 115 can be reduced by the Hough transform in the process of step S704 described later. Therefore, through the process of step S702, the control unit 24 determines whether there are point groups 112 and 113 such as those shown in Figure 32 that are located near the point group 111 and the point group 115. Figure 32 As shown, the point group 112 and the point group 113 are located near the point group 111 and the point group 115. If the point group 112 and the point group 113 are located near the point group 111 and the point group 115, the accuracy of detecting the point group 111 and the point group 115 can be reduced by the Hough transform in the process of step S704 described later. Therefore, through the process of step S702, the control unit 24 determines whether there are point groups 112 and 113 such as those shown in Figure 32 that are located near the point group 111 and the point group 115.

[0217] Specifically, in the process of step S702, the control unit 24 determines whether there is a point group that is substantially parallel to the u direction. As shown in Figure 32 , the point group 112 and the point group 113 located near the point group 111 and the point group 115 are the parallax obtained from the images 122 and 123 corresponding to other vehicles as shown in Figure 33 . The other vehicles include parts that are substantially parallel to the width direction of the road surface. As shown in Figure 32 , since the other vehicles include parts that are substantially parallel to the width direction of the road surface, the point group 112 and the point group 113 can be in a shape that is substantially parallel to the u direction. Therefore, by determining whether there is a point group that is substantially parallel to the u direction, it can be determined whether there are point groups 112 and 113 such as those shown in Figure 32 . Figure 32 As shown, the point group 112 and the point group 113 located near the point group 111 and the point group 115 are the parallax obtained from the images 122 and 123 corresponding to other vehicles as shown in Figure 33 . The other vehicles include parts that are substantially parallel to the width direction of the road surface. As shown in Figure 32 , since the other vehicles include parts that are substantially parallel to the width direction of the road surface, the point group 112 and the point group 113 can be in a shape that is substantially parallel to the u direction. Therefore, by determining whether there is a point group that is substantially parallel to the u direction, it can be determined whether there are point groups 112 and 113 such as those shown in Figure 32 . Figure 33 As shown, the point group 112 and the point group 113 located near the point group 111 and the point group 115 are the parallax obtained from the images 122 and 123 corresponding to other vehicles as shown in Figure 33 . The other vehicles include parts that are substantially parallel to the width direction of the road surface. As shown in Figure 32 , since the other vehicles include parts that are substantially parallel to the width direction of the road surface, the point group 112 and the point group 113 can be in a shape that is substantially parallel to the u direction. Therefore, by determining whether there is a point group that is substantially parallel to the u direction, it can be determined whether there are point groups 112 and 113 such as those shown in Figure 32 . Figure 32 As shown, since the other vehicles include parts that are substantially parallel to the width direction of the road surface, the point group 112 and the point group 113 can be in a shape that is substantially parallel to the u direction. Therefore, by determining whether there is a point group that is substantially parallel to the u direction, it can be determined whether there are point groups 112 and 113 such as those shown in Figure 32 . Figure 32 As shown, since the other vehicles include parts that are substantially parallel to the width direction of the road surface, the point group 112 and the point group 113 can be in a shape that is substantially parallel to the u direction. Therefore, by determining whether there is a point group that is substantially parallel to the u direction, it can be determined whether there are point groups 112 and 113 such as those shown in Figure 32 .

[0218] As an example of the process of step S702, first, the control unit 24 scans along the u direction of the UD map. For example, the control unit 24 scans from the negative direction side of the u axis of the UD map 110 as shown in Figure 32 to the positive direction side of the u axis. While scanning along the u direction of the UD map, the control unit 24 determines whether there are points arranged continuously within a specified range along the u direction. The specified range can be set based on the length along the u direction of the image of the vehicle (for example, the images 122 and 123 shown in Figure 33 ). When the control unit 24 determines that there are points arranged continuously within a specified range along the u direction, it determines that there is a point group that is substantially parallel to the u direction. The control unit 24 detects the parallax points arranged continuously within a specified interval along the u direction as a point group that is substantially parallel to the u direction. For example, the control unit 24 detects the point group 112 and the point group 113 shown in Figure 32 as a point group that is substantially parallel to the u direction. Figure 32 As an example of the process of step S702, first, the control unit 24 scans along the u direction of the UD map. For example, the control unit 24 scans from the negative direction side of the u axis of the UD map 110 as shown in Figure 32 to the positive direction side of the u axis. While scanning along the u direction of the UD map, the control unit 24 determines whether there are points arranged continuously within a specified range along the u direction. The specified range can be set based on the length along the u direction of the image of the vehicle (for example, the images 122 and 123 shown in Figure 33 ). When the control unit 24 determines that there are points arranged continuously within a specified range along the u direction, it determines that there is a point group that is substantially parallel to the u direction. The control unit 24 detects the parallax points arranged continuously within a specified interval along the u direction as a point group that is substantially parallel to the u direction. For example, the control unit 24 detects the point group 112 and the point group 113 shown in Figure 32 as a point group that is substantially parallel to the u direction. Figure 33 As an example of the process of step S702, first, the control unit 24 scans along the u direction of the UD map. For example, the control unit 24 scans from the negative direction side of the u axis of the UD map 110 as shown in Figure 32 to the positive direction side of the u axis. While scanning along the u direction of the UD map, the control unit 24 determines whether there are points arranged continuously within a specified range along the u direction. The specified range can be set based on the length along the u direction of the image of the vehicle (for example, the images 122 and 123 shown in Figure 33 ). When the control unit 24 determines that there are points arranged continuously within a specified range along the u direction, it determines that there is a point group that is substantially parallel to the u direction. The control unit 24 detects the parallax points arranged continuously within a specified interval along the u direction as a point group that is substantially parallel to the u direction. For example, the control unit 24 detects the point group 112 and the point group 113 shown in Figure 32 as a point group that is substantially parallel to the u direction. Figure 32 While scanning along the u direction of the UD map, the control unit 24 determines whether there are points arranged continuously within a specified range along the u direction. The specified range can be set based on the length along the u direction of the image of the vehicle (for example, the images 122 and 123 shown in Figure 33 ). When the control unit 24 determines that there are points arranged continuously within a specified range along the u direction, it determines that there is a point group that is substantially parallel to the u direction. The control unit 24 detects the parallax points arranged continuously within a specified interval along the u direction as a point group that is substantially parallel to the u direction. For example, the control unit 24 detects the point group 112 and the point group 113 shown in Figure 32 as a point group that is substantially parallel to the u direction.

[0219] In the process of step S702, when the control unit 24 determines that there is a point group substantially parallel to the u direction (step S702: Yes), it proceeds to the process of step S703. On the other hand, when the control unit 24 determines that there is no point group substantially parallel to the u direction (step S702: No), it proceeds to the process of step S704.

[0220] In the process of step S703, the control unit 24 removes the point group substantially parallel to the detected u direction from the UD diagram. Instead of removing the point group substantially parallel to the u direction from the UD diagram, the control unit 24 may exclude only the point group substantially parallel to the u direction among the coordinate points of the UD diagram from the application of the Hough transform in the process of step S704 described later. In Figure 34 FIG. 110 shows a UD diagram from which the point group substantially parallel to the u direction has been removed. In Figure 34 In the UD diagram 110 shown, the Figure 32 shown point group 112 and point group 113 are removed.

[0221] In the process of step S704, the control unit 24 detects a straight line by applying the Hough transform to the points included in the UD diagram. Among them, by transforming the UD diagram into a coordinate system of the actual space composed of the x-z coordinates and applying the Hough transform to the transformed coordinate system of the actual space, a straight line can also be detected. However, if the baseline length B between the first camera 11 and the second camera 12 is short, the interval between the coordinate points included in the coordinate system of the actual space may be sparser than the interval between the coordinate points included in the UD diagram. If the interval between the coordinate points is sparse, it may not be possible to detect a straight line with high accuracy by the Hough transform. In the present embodiment, by applying the Hough transform to the UD diagram, even if the baseline length B between the first camera 11 and the second camera 12 is short, a straight line can be detected with high accuracy.

[0222] Refer to Figure 35 and Figure 36 to illustrate an example of the process of step S704.

[0223] In Figure 35 a part of the UD diagram is shown. Points 131, 132, 133, and 134 are coordinate points on the UD diagram. Taking point 131 as an example to illustrate the Hough transform. The uv coordinates of point 131 on the UD diagram are coordinates (u 131 , d 131 ). The straight line L1 passing through point 131 can be defined infinitely. For example, the length of the normal from the origin (0, 0) of the uv coordinates to the straight line L1-1 is length r. This normal is inclined at an angle θ from the u axis to the positive direction of the d axis. The control unit 24 obtains the following mathematical formula (3) as the general formula of the straight line L1 passing through point 131 by setting the length r and the angle θ as variables.

[0224] r = u 131 ×cosθ + d 131 ×sinθ (3)

[0225] The control unit 24 projects the equation of the straight line L1 (mathematical formula (3)) onto the rθ plane, which is a Hough space, as shown in Figure 36 the figure. Figure 36 The horizontal axis of the rθ plane shown in the figure is the r-axis. Figure 36 The vertical axis of the rθ plane shown in the figure is the θ-axis. Figure 36 The curve 131L shown in the figure is the curve represented by the mathematical formula (3). The curve 131L is represented as a sine curve in the rθ plane. Similarly to the point 131, the control unit 24 obtains the equation of the straight line passing through the Figure 35 points 132 to 134 shown in the figure. Similarly to the point 131, the control unit 24 projects the equation of the straight line passing through the obtained points 132 to 134 onto the rθ plane, which is a Hough space, as shown in Figure 36 the figure. Figure 36 The curves 132L to 134L shown in the figure respectively correspond to the straight lines passing through the Figure 35 points 132 to 134 obtained as shown in the figure. As shown in Figure 36 the figure, the curves 131L to 134L may intersect at the point P L . The control unit 24 obtains the rθ coordinates of the point P L (coordinates (θ L , r L )). Based on the coordinates (θ L , r L ) of the point P L , the control unit 24 detects the equation of the straight line passing through the Figure 35 points 131 to 134 shown in the figure. The control unit 24 detects the following mathematical formula (4) as the straight line passing through the Figure 35 points 131 to 134 shown in the figure.

[0226] r L = u×cosθ L + d×sinθ L (4)

[0227] By performing the process of step S704, as shown in Figure 34 the figure, the control unit 24 can detect the straight line 111L corresponding to the point group 111 and the straight line 115L corresponding to the point group 115.

[0228] In addition, as shown in Figure 33 the figure, the images 121 and 125 corresponding to the parallel objects extend toward the vanishing point 120 VP . As shown in Figure 34 the figure, since the images 121 and 125 extend toward the vanishing point 120VP extends, so the point group 111 and the point group 115 also extend towards the vanishing point 120 VP corresponding vanishing point 110 VP extends. Since the point group 111 and the point group 115 also extend towards the vanishing point 110 VP extends, the straight line 111L and the straight line 115L can also extend towards the vanishing point 110 VP extends.

[0229] Therefore, in the process of step S704, when the control unit 24 obtains the formula of the straight line passing through the points on the UD diagram, it can obtain the formula of the straight line passing through the specified range based on the vanishing point among the straight lines passing through this point. For example, when the control unit 24 obtains the formula of the straight line L1 passing through the point 131 as shown Figure 35 in the formula of the straight line L1 that infinitely exists passing through the point 131, it can obtain the formula of the straight line passing through the point 131 and the specified range Δu VP The specified range Δu VP is the range based on the vanishing point. The specified range Δu VP includes the point 135. The point 135 can be the vanishing point when the traveling direction of the moving body 30 is straight. By making the parallax at infinity zero, the d coordinate of the point 135 can be zero. Since when the traveling direction of the moving body 30 is straight, the u coordinate of the vanishing point is half of the maximum coordinate of the u coordinate, the u coordinate (coordinate u VP ) of the point 135 can be half of the maximum coordinate of the u coordinate. The specified range Δu VP can be appropriately set based on the offset between the vanishing point 110 as shown Figure 34 and the point 135 when the moving body 30 is traveling on a curve. Through such processing, as VP shown, in the rθ plane of the Hough space, the range of the θ axis for drawing the curves 131L to 134L can be reduced to the range Δθ Figure 36 . By reducing the range of the θ axis for drawing the curves 131L to 134L to the range Δθ VP , the computational amount of the Hough transform can be reduced. By reducing the computational amount of the Hough transform, the processing of the Hough transform can be speeded up. VP In the process of step S705, the control unit 24 determines whether the length of the straight line detected through the process of step S704 exceeds the specified length. The specified length can be appropriately set based on the length of the building, i.e., the parallel object, arranged along the road surface. When the control unit 24 determines that the length of the straight line is below the specified length (step S705: No), it returns to

[0230] In the process of step S705, the control unit 24 determines whether the length of the straight line detected through the process of step S704 exceeds the specified length. The specified length can be appropriately set based on the length of the building, i.e., the parallel object, arranged along the road surface. When the control unit 24 determines that the length of the straight line is below the specified length (step S705: No), it returns to Figure 5The processing of step S107 shown. On the other hand, when the control unit 24 determines that the length of the straight line exceeds a specified length (step S705: YES), it attaches a parallel object mark to the point group corresponding to the straight line (step S706). For example, the control unit 24 attaches a parallel object mark to Figure 34 the point group 111 and the point group 115 shown. After executing the processing of step S706, the control unit 24 proceeds to Figure 5 the processing of step S107 shown.

[0231] In the processing of step S107, the control unit 24 executes a recovery process. The detailed content of the processing of step S107 is shown in the flowchart Figure 37 shown.

[0232] In the processing of step S801, the control unit 24 performs the same or similar processing as the processing of Figure 31 step S701 shown, and generates or acquires a UD map. When generating a UD map in the processing of step S701 shown, the control unit 24 can acquire the UD map generated in the processing of Figure 31 step S701 shown. Figure 31

[0233] Figure 38 UD map 140 is shown in Figure 38 . UD map 140 is a magnified view of a part of Figure 34 UD map 110 shown. The horizontal axis of UD map 140 corresponds to the u axis. The vertical axis of UD map 140 corresponds to the d axis representing the magnitude of the parallax. In Figure 38 , UD map 140 is shown as an image. The pixels of UD map 140 represent the parallax. The pixels marked with hatched lines are the pixels representing the first parallax. The pixels representing the first parallax, that is, the coordinates corresponding to the first parallax are also referred to as "first parallax points". The pixels marked with dots are the pixels representing the second parallax. The pixels representing the second parallax, that is, the coordinates corresponding to the second parallax are also referred to as "second parallax points". UD map 140 includes: a first parallax point 141, a first parallax point 142, and a second parallax point 143. UD map 140 is a map generated based on a second parallax image including a parallax image corresponding to the rear of the vehicle. Since the rear glass or the like is located in the central part of the rear of the vehicle, the feature amount on the stereo image is small. In the central part of the rear of the vehicle, since the feature amount on the stereo image is small, compared with the parallax of other parts, as Figure 19 the parallax image 44 shown, sometimes the information of the obtainable parallax becomes less. For example, as Figure 38 shown, there is a second parallax point 143 between the first parallax point 141 and the first parallax point 142.

[0234] In the process of step S802, in the u direction of the UD map, the control unit 24 determines whether a second parallax point sandwiched between the first parallax points exists beyond a specified range. For example, the control unit 24 scans from the negative direction side of the u-axis of the UD map to the positive direction side of the u-axis. By scanning along the u direction, the control unit 24 determines whether a second parallax point sandwiched between the first parallax points exists beyond a specified range in the u direction of the UD map. The specified range can be appropriately set based on the width of the rear of the vehicle in the actual space (e.g., 1 m). When a second parallax point sandwiched between the first parallax points exists beyond the specified range, these second parallax points are likely to be parallaxes corresponding to different objects such as, for example, different parallel vehicles. In contrast, when a second parallax point sandwiched between the first parallax points exists within the specified range, these second parallax points are likely to be parallaxes corresponding to the same object. In Figure 38 In the u direction of the UD map 140 shown, the second parallax point 143 sandwiched between the first parallax point 141 and the first parallax point 142 exists within the specified range. In the u direction of the UD map 140, the control unit 24 does not determine that the second parallax point 143 sandwiched between the first parallax point 141 and the first parallax point 142 exists beyond the specified range.

[0235] In the process of step S802, in the u direction of the UD map, when the control unit 24 determines that a second parallax point sandwiched between the first parallax points exists beyond the specified range (step S802: Yes), it proceeds to the process of step S805. On the other hand, in the u direction of the UD map, when the control unit 24 does not determine that a second parallax point between the first parallax points exists beyond the specified range (step S802: No), it proceeds to the process of step S803.

[0236] In the process of step S803, the control unit 24 acquires the first parallaxes respectively corresponding to the two first parallax points sandwiching the second parallax point. For example, the control unit 24 acquires the first parallax corresponding to Figure 38 the first parallax point 141 shown and the first parallax corresponding to the first parallax point 142. Further, in the process of step S802, the control unit 24 determines whether the difference in the actual space height corresponding to each of the two acquired first parallaxes is within a specified height. For example, the control unit 24 determines whether the difference between the actual space height corresponding to the first parallax of the first parallax point 141 shown and the actual space height corresponding to the first parallax of the first parallax point 142 is below the specified height. The specified height can be appropriately set based on the height of the vehicle in the actual space (e.g., 1 m). Figure 38 In the u direction of the UD map 140 shown, the control unit 24 determines whether the difference between the actual space height corresponding to the first parallax of the first parallax point 141 and the actual space height corresponding to the first parallax of the first parallax point 142 is below the specified height.

[0237] In the process of step S803, when the control unit 24 determines that the difference in the height in the actual space corresponding to each of the two acquired first disparities is within a specified height (step S803: Yes), it proceeds to the process of step S804. In the process of step S804, the control unit 24 attaches a recovery flag to the second disparity points that exceed the specified range and are sandwiched between the first disparity points. As described above, the second disparity points represent the ud coordinates of the second disparity. That is to say, the process of step S804 can also be referred to as the process of attaching a recovery flag to the second disparity and the u coordinate corresponding to the second disparity. Figure 39 UD map 140 with a recovery flag attached. Figure 39 The pixels marked with thick shaded lines shown are the second disparity points with a recovery flag attached.

[0238] In the process of step S803, when the control unit 24 determines that the difference in the height in the actual space corresponding to each of the two acquired first disparities exceeds the specified height (step S803: No), it proceeds to the process of step S805.

[0239] The processes after step S805 are to determine which of the first candidate height calculated in the process of step S619 shown in Figure 25 and the second candidate height calculated in the process of step S622 shown in Figure 25 is the height of the object. Before explaining the processes after step S805, refer to Figure 40 and Figure 41 to illustrate an example of the second disparity image.

[0240] In Figure 40 the second disparity image 150 is shown. In Figure 41 the second disparity image 160 is shown. In Figure 40 and Figure 41 a part of the second disparity image 150 and the second disparity image 160 are shown. In the second disparity image 150 and the second disparity image 160, the white disparity pixels are the pixels that do not contain disparity information. The disparity pixels marked with thicker shaded lines are the pixels that contain disparity information. The disparity pixels marked with thinner shaded lines are the disparity pixels used to calculate the second candidate height in the process of step S622 shown in Figure 25 . In the process of step S622 shown in Figure 25 , the coordinates of the disparity pixels marked with thinner shaded lines are associated with the second candidate height as candidate coordinates. The disparity pixels marked with thick dots are the disparity pixels used to calculate the first candidate height in the process of step S619 shown in Figure 25 . In Figure 25In the process of step S619 shown, the coordinates of the parallax pixels marked with thick dots are associated with the first candidate height as the first coordinates. The parallax pixels marked with thin dots are used to calculate Figure 24 the height of the object in the process of step S616 shown. In Figure 24 the process of step S616 shown, the coordinates of the parallax pixels marked with thin dots are associated with the height of the object as the first coordinates.

[0241] Similar to [[ID=2 the second parallax image 100 shown, ​ the second parallax image 150 shown is an image generated based on ​ the image 107 of the rear of the truck shown. The white parallax pixels are located in the central part of the second parallax image 150. That is, in the second parallax image 150, the parallax of the parallax pixels in the central part is not calculated. In the second parallax image 150, the parallax of the parallax pixels surrounding the central part is calculated.

[0242] ​ The second parallax image 150 shown includes parallax pixels 151, 152, 153, 154. The parallax pixel 151 is a parallax pixel for calculating the second candidate height. The coordinates of the parallax pixel 151 are associated with the second candidate height as candidate coordinates. The parallax pixels 152 and 153 are parallax pixels for calculating the height of the object. Each of the coordinates of the parallax pixels 152 and 153 is associated with the height of the object as the first coordinates. The parallax pixel 154 is a parallax pixel for calculating the first candidate height. The coordinates of the parallax pixel 154 are associated with the first candidate height as the first coordinates.

[0243] Similar to ​ the parallax pixel 105 shown, the parallax pixel 151 corresponds to ​ the upper image 107b of the image 107 of the rear of the truck shown. Similar to ​ the parallax pixel 103 shown, the parallax pixel 154 corresponds to ​ the lower image 107a of the image 107 of the rear of the truck shown. In ​ the structure shown, it is necessary to obtain the second candidate height based on the parallax pixel 151 and the first candidate height based on the parallax pixel 154, and take the second candidate height based on the parallax pixel 151 as the height of the object, that is, ​ the height of the truck shown.

[0244] ​ The second parallax image 160 shown is based on ​The parallax image generated from the first image 170 shown. The first image 170 includes an image 171 of the upper part of another vehicle and an image 172 corresponding to a street tree. The parallax generated from the image 171 is approximately equal to the parallax generated from the image 172. The image 172 includes a partial image 172a and a partial image 172b. The partial image 172a and the image 171 of the vehicle are located in the central part of the first image 170 in the u direction. The u coordinate of the partial image 172a is the same as a part of the u coordinate of the image 171 of the other vehicle. The partial image 172a is closer to the negative direction side of the v axis than the image 171 of the other vehicle. The partial image 172b is closer to the negative direction side of the u axis than the partial image 172a and the image 171.

[0245] ​ The second parallax image 160 shown includes: a parallax pixel 161, a parallax pixel 162, and a parallax pixel 163. The parallax pixel 161 is a parallax pixel for calculating a second candidate height. Calculate according to the coordinates of the parallax pixel 161 ​ The height of the partial image 172a of the street tree shown as the second candidate height. The coordinates of the parallax pixel 161 are associated with the second candidate height as candidate coordinates. The parallax pixel 162 is a parallax pixel for calculating the height of an object. Calculate according to the coordinates of the parallax pixel 162 ​ The height of the partial image 172b of the street tree shown as the height of the object. The coordinates of the parallax pixel 162 are associated with the height of the object as candidate coordinates. The parallax pixel 163 is a parallax pixel for calculating a first candidate height. Calculate according to the coordinates of the parallax pixel 163 ​ The height of the image 171 of the vehicle shown as the first candidate height. The coordinates of the parallax pixel 163 are associated with the first candidate height as the first coordinates.

[0246] In ​ In the structure shown, it is necessary to obtain the first candidate height based on the parallax pixel 163 and the second candidate height based on the parallax pixel 161, and take the first candidate height based on the parallax pixel 163 as the height of the object, that is ​ The height of the image 171 of the vehicle shown.

[0247] In the processing of steps S805 to S807, the control unit 24 determines which of the first candidate height and the second candidate height to obtain as the height of the object based on the parallax corresponding to the two coordinates sandwiching the candidate coordinates.

[0248] Specifically, in the processing of step S805, the control unit 24 determines whether there are two first coordinates sandwiching the candidate coordinates in the u direction by scanning the second parallax image along the u direction. As described above, as ​As shown, a parallax approximately equal to the object parallax corresponds to the first coordinate. That is to say, the process of step S805 can also be referred to as determining whether the parallax corresponding to the two coordinates sandwiching the candidate coordinate in the u direction is the object parallax.

[0249] For example, in ​ in the u direction of the second parallax image 150 shown, the candidate coordinate of the parallax pixel 151 is sandwiched by the first coordinate of the parallax pixel 152 and the first coordinate of the parallax pixel 153. In ​ in the structure shown, the control unit 24 determines that there are the first coordinates of the parallax pixel 152 and the first coordinate of the parallax pixel 153 sandwiching the candidate coordinate of the parallax pixel 151.

[0250] For example, in ​ in the u direction of the second parallax image 160 shown, the first coordinate of the parallax pixel 162 is located on the negative u-axis side of the candidate coordinate of the parallax pixel 161. On the other hand, the first coordinate is not located on the positive u-axis side of the candidate coordinate of the parallax pixel 161. In ​ in the structure shown, the control unit 24 determines that there are no two first coordinates sandwiching the candidate coordinate of the parallax pixel 161.

[0251] In the process of step S805, when the control unit 24 determines that there are two first coordinates sandwiching the candidate coordinate in the u direction of the second parallax image (step S805: Yes), it proceeds to the process of step S806. On the other hand, when the control unit 24 determines that there are no two first coordinates sandwiching the candidate coordinate in the u direction of the second parallax image (step S805: No), it proceeds to the process of step S807.

[0252] In the process of step S806, the control unit 24 determines to obtain the second candidate height among the first candidate height and the second candidate height as the height of the object. Through the processes of steps S805 and S806, when the control unit 24 determines that the parallax corresponding to the two coordinates sandwiching the candidate coordinate is the object parallax, it obtains the second candidate height as the height of the object. For example, the control unit 24 determines to obtain the second candidate height based on the coordinate of the parallax pixel 151 among the first candidate height based on the coordinate of the parallax pixel 154 shown in ​ and the second candidate height based on the coordinate of the parallax pixel 151 as the height of the object. In ​ shows the parallax pixels used to determine the height of the object in the second parallax image 150 shown in ​ . In ​ , the height of the object is calculated based on the coordinates of the parallax pixels marked with fine hatching. As shown in ​As shown, obtain the second candidate height based on the parallax pixel 151 and the first candidate height based on the parallax pixel 154, and take the second candidate height based on the parallax pixel 151 as the height of the object, that is ​ the height of the truck shown.

[0253] In the process of step S807, the control unit 24 determines to obtain the first candidate height among the first candidate height and the second candidate height as the height of the object. Through the processes of steps S805 and S807, when the control unit 24 does not determine that the parallaxes corresponding to the two coordinates sandwiching the candidate coordinates are the target parallaxes, it determines to obtain the first candidate height as the height of the object. For example, the control unit 24 determines to obtain ​ the first candidate height based on the coordinates of the parallax pixel 163 shown and the second candidate height based on the coordinates of the parallax pixel 161, and take the first candidate height based on the coordinates of the parallax pixel 163 as the height of the object. In ​ It shows the parallax pixels used to determine the height of the object in the ​ second parallax image 160 shown. In ​ it, the height of the object is calculated based on the coordinates of the parallax pixels marked with fine shaded lines. As ​ shown, obtain the first candidate height based on the parallax pixel 163 and the second candidate height based on the parallax pixel 161, and take the first candidate height based on the parallax pixel 163 as the height of the object, that is ​ the height of the vehicle image 171 shown.

[0254] After executing the processes of steps S806 and S807, the control unit 24 proceeds to ​ the process of step S108 shown.

[0255] In the process of step S108, the control unit 24 performs the determination process of the representative parallax. The detailed content of the process of step S108 is shown in the ​ flowchart shown.

[0256] In the process of step S901, the control unit 24 performs the same or similar process as the ​ process of step S701 shown, and generates a UD map. When generating a UD map in the process of step S701 shown, the control unit 24 can obtain the UD map generated in the ​ process of step S701 shown. ​ In the process of step S902, the control unit 24, in each u coordinate of the UD map, selects from the first parallax and through

[0257] In the process of step S902, the control unit 24, in each u coordinate of the UD map, selects from the first parallax and through ​The process of step S804 shown obtains a representative disparity from the second disparity with a recovery flag attached. The control unit 24 obtains a representative disparity from the first disparity without a parallel object flag attached by the process of step S706 shown and the second disparity with a recovery flag attached. ​ The process of step S706 shown obtains a representative disparity from the first disparity without a parallel object flag attached and the second disparity with a recovery flag attached.

[0258] Among them, when the distance from the stereo camera 10 to the object is relatively close, compared with the case where the distance from the stereo camera 10 to the object is relatively far, the number of pixels occupied by the object on the stereo image may be larger. In addition, the farther the object is from the stereo camera 10, the more affected it is by noise and the like, resulting in a deterioration in the accuracy of detecting the disparity corresponding to the object. That is to say, the closer the object is to the stereo camera 10, the higher the accuracy of detecting the disparity corresponding to the object.

[0259] Therefore, in the process of step S902, at each u coordinate of the UD map, the control unit 24 obtains the maximum disparity corresponding to the object close to the stereo camera 10, that is, the representative disparity, from the first disparity and the second disparity with a recovery flag attached. Further, the control unit 24 can obtain the disparity corresponding to the object close to the stereo camera 10, that is, the object with a height greater than the minimum value (for example, 50 cm) of the object that is the detection target of the object detection device 20, as the representative disparity.

[0260] In ​ an example of the obtained representative disparity corresponding to the u direction is shown. In the example shown in ​ , the deviation of the representative disparity with respect to each u coordinate is relatively large. If the deviation of the representative disparity with respect to each u coordinate is relatively large, it is possible to increase the amount of computation of the grouping process in the process of step S109.

[0261] Therefore, in the process of step S903, the control unit 24 can average the representative disparities within a specified range. By averaging the representative disparities within a specified range, the amount of computation of the grouping process in the process of step S109 can be reduced. The specified range can be appropriately set in consideration of the computational load such as the process of step S109 described later. When performing averaging, the control unit 24 can obtain the representative disparity of the central value of the specified range. The control unit 24 can calculate the average value of the representative disparities by removing the representative disparities that deviate by a specified ratio (for example, 5%) from the extracted central values within the specified range. In ​ an example of the averaged representative disparity corresponding to the u direction is shown. After executing the process of step S903, the control unit 24 proceeds to the process of step S109 shown in ​ .

[0262] The control unit 24 can perform after ​After the processing of steps S104 to S108 shown above, i.e., the third processing, for each specified range of the horizontal coordinate (u coordinate) including one or more coordinates, the representative disparity d e is stored in the memory 23 in association with the height information of the object. As ​ shown, the multiple representative disparities d e stored in the memory 23 can be expressed as the distribution of a point group on a two-dimensional space (u-d coordinate space) with the u coordinate and the disparity d as the horizontal axis and the vertical axis, respectively.

[0263] In the processing of step S109, the control unit 24 performs the processing of detecting an object (fourth processing) by extracting a set (group) of representative disparities d e by transforming the information of the representative disparity d e in the u-d coordinate space into the coordinate system of the actual space composed of the x-z coordinates. Refer to ​ and ​ for an example of the processing of step S109.

[0264] ​ is an example of the structure in the actual space. In ​ , a moving body 30 equipped with the object detection system 1 and other vehicles 42A traveling on the road surface 41A are shown. In ​ , the moving body 30 is a vehicle.

[0265] The control unit 24 of the object detection device 20 mounted on the moving body 30 transforms the multiple representative disparities d ​ in the u-d coordinate space shown in e into the point group in the actual space (x-z coordinate space) shown in ​ . In ​ , each point representing the representative disparity d e in the u-d coordinate space is displayed as a point in the x-z coordinate space. The control unit 24 extracts a set of point groups based on the distribution of the point group. The control unit 24 extracts the set of point groups by aggregating multiple points that are close according to a specified condition. The set of point groups represents a set (group) of representative disparities d e .

[0266] When the object has a surface parallel to the baseline length direction of the stereo camera 10, in the x-z coordinate space, the point group is arranged along the x direction. When there is a set 180 of point groups arranged along the x direction in the x-z coordinate space, the control unit 24 can identify it as an object. In ​ , the concentration 180 of the point group corresponds to the rear surface of the body of the other vehicle 42A shown in ​ .

[0267] When the object is a parallel object, the point group is arranged in the z direction in the x-z coordinate space. When there is a set 181 of point groups arranged in the z direction in the x-z coordinate space, the control unit 24 can recognize it as a parallel object. As described above, as an example of a parallel object, roadside buildings such as guardrails and sound insulation walls on highways can be cited, or, ​ the side surface of another vehicle 42A shown in the figure, etc. The set 181 of point groups arranged in the z direction in the x-z coordinate space corresponds to an object arranged parallel to the traveling direction of the moving body 30, or a plane parallel to the traveling direction of the moving body 30 of the object. The control unit 24 can remove the set 181 of point groups arranged in the z direction from the objects to be detected in the object detection process. Among them, the parallax corresponding to the parallel object detected in the above clustering process is regarded as ​ the representative parallax d in the process of step S902 shown in the figure e a parallax other than the object of the acquisition process. However, in the clustering process, there may be a case where the parallax corresponding to the parallel object cannot be completely detected. In this case, in the x-z coordinate space, there may be a parallax corresponding to the parallel object, such as the set 181 of point groups. Even when there is a parallax corresponding to the parallel object like the set 181 of point groups in the x-z coordinate space, in the process of step S109, the parallax corresponding to the parallel object can be excluded from the objects to be detected in the object detection process.

[0268] The control unit 24 can detect the width of the object according to the width of the set 180 of point groups recognized as the object arranged in the x direction. The control unit 24 can determine the height of the object based on the height information associated with the representative parallax d obtained through the ​ process shown in the figure e Therefore, the control unit 24 can recognize the position, lateral width, and height of the recognized object in the x-z coordinate space.

[0269] In the process of step S110, the control unit 24 can output the information on the position, lateral width, and height of the object recognized through the process of step S109 to other devices in the moving body 30 through the output unit 22. For example, the control unit 24 can output this information to the display device in the moving body 30. As ​ shown in the figure, the display device in the moving body 30 can display a detection frame 182 surrounding the image corresponding to another vehicle in the image of the first camera 11 or the second camera 12 based on the information obtained from the object detection device 20. The detection frame 182 indicates the position of the detected object and the range occupied in the image.

[0270] As described above, the object detection device 20 of the present invention can achieve high processing speed and high-precision object detection. That is, the object detection device 20 of the present invention and the object detection method of the present invention can improve the performance of detecting objects. In addition, the object detection device 20 does not limit the object to be detected to a specific type of object. The object detection device 20 can detect all objects existing on the road surface. The control unit 24 of the object detection device 20 can perform the first process, the second process, the third process, and the fourth process without using the information of images other than the first disparity image captured by the stereo camera 10. Therefore, in addition to the processing of the first disparity image and the second disparity image, the object detection device 20 may not perform the process of separately identifying an object based on the captured image. Therefore, the object detection device 20 of the present invention can reduce the processing load of the control unit 24 during object recognition. This does not exclude the case where the object detection device 20 of the present invention is combined with image processing of images directly obtained from the first camera 11 or the second camera 12. The object detection device 20 can also be combined with image processing techniques such as template matching.

[0271] In the description of the processes performed by the control unit 24 described above, for the purpose of facilitating the understanding of the present invention, the processes including the determination and operations using various images are described. The processes using these images may not include the process of actually drawing images. The processes substantially the same in content as the processes using these images are performed through the information processing inside the control unit 24.

[0272] For the embodiments of the present invention, the description is based on the respective drawings and examples, but it should be noted that those skilled in the art can easily make various deformations or modifications based on the present invention. Therefore, it should be noted that these deformations or modifications are included in the scope of the present invention. For example, the functions and the like included in each component or each step can be reconfigured in a logically non-contradictory manner, and multiple components or steps can be combined into one or divided. The embodiments of the present invention are described centering on the device, but the embodiments of the present invention can also be implemented as a method including the steps performed by each component of the device. The embodiments of the present invention can be implemented as a method, a program, or a storage medium recording the program executed by a processor included in the device. It should be understood that these are also included in the scope of the present invention.

[0273] In the present invention, descriptions such as "first" and "second" are identifiers for distinguishing the structures. The structures distinguished in the descriptions such as "first" and "second" in the present invention can exchange the numbers in the structures. For example, the first lens can exchange "first" and "second" as identifiers with the second lens. The exchange of the identifiers is carried out simultaneously. The structures can also be distinguished after the identifiers are exchanged. The identifiers can be deleted. The structures with the identifiers deleted are distinguished by reference numerals. The descriptions of the identifiers such as "first" and "second" in the present invention should not be used to interpret the order of the structures, nor can they be used as the basis for the existence of identifiers with smaller numbers.

[0274] In the present invention, the x-direction, y-direction, and z-direction are set for convenience of explanation and can be replaced with each other. An orthogonal coordinate system with the x-direction, y-direction, and z-direction as the respective axis directions is used to describe the structure of the present invention. The positional relationship of the respective structures of the present invention is not limited to an orthogonal relationship. The u-coordinate and v-coordinate representing the coordinates of the image are set for convenience of explanation and can be replaced with each other. The origin and direction of the u-coordinate and v-coordinate are not limited to the content of the present invention.

[0275] In the above embodiment, the first camera 11 and the second camera 12 of the stereo camera 10 are arranged and configured in the x-direction. The arrangement of the first camera 11 and the second camera 12 is not limited to this. The first camera 11 and the second camera 12 can be arranged and configured in the direction perpendicular to the road surface (y-direction) or in a direction inclined with respect to the road surface 41A. The number of cameras constituting the stereo camera 10 is not limited to two. The stereo camera 10 can include three or more cameras. For example, it is also possible to use a total of four cameras, two cameras arranged horizontally on the road surface and two cameras arranged vertically, to obtain more accurate distance information.

[0276] In the above embodiment, the stereo camera 10 and the object detection device 20 are mounted on the moving body 30. The stereo camera 10 and the object detection device 20 are not limited to being mounted on the moving body 30. For example, the stereo camera 10 and the object detection device 20 can also be configured to be mounted on a roadside unit provided at an intersection or the like to capture an image including the road surface. For example, the roadside unit can provide the following information: detecting a first vehicle approaching from one of the roads intersecting at an intersection and notifying a second vehicle traveling on the other road and approaching of the approach of the first vehicle.

[0277] Explanation of reference numerals:

[0278] 1, 1A Object detection system

[0279] 10 Stereo camera

[0280] 11 First camera

[0281] 12 Second camera

[0282] 20 Object detection device

[0283] 21 Acquisition unit

[0284] 22 Output unit

[0285] 23 Memory

[0286] 24 Control unit

[0287] 25 Generation device

[0288] 30 Moving body

[0289] 40 First parallax image

[0290] 41, 42, 43, 44, 71, 72, 73, 74, 75, 76, 81, 82, 101 Parallax images

[0291] 41A Road surface

[0292] 42A Vehicle

[0293] 42a, 83, 84, 85, 102, 103, 104, 105, 151, 152, 153, 154, 161, 162, 163 Parallax pixels

[0294] 45 Frame line

[0295] 51 Graphic

[0296] 52 First straight line

[0297] 53 Approximation start point

[0298] 54 Candidate straight line

[0299] 55 Second straight line

[0300] 60, 70, 80, 100, 150, 160 Second parallax images

[0301] 61 Partial area

[0302] 61-1, 61 Partial area

[0303] 62 Area

[0304] 90, 106, 120, 170 First image

[0305] 91, 92, 107, 121, 122, 123, 124, 125, 171, 172 Images

[0306] Partial area of 101a

[0307] Lower image of 107a

[0308] Upper image of 107b

[0309] 110, 140UD diagram

[0310] 110 VP , 120 VP Vanishing point

[0311] Point groups of 111, 112, 113, 114, 115, 180, 181

[0312] Lines of 111L, 115L

[0313] Points of 131, 132, 133, 134

[0314] Curves of 131L, 132L, 133L, 134L, 135L

[0315] First parallax points of 141, 142

[0316] Second parallax point of 143

[0317] Partial images of 172a, 172b

[0318] Detection frame of 182

Claims

1. An object detection device, wherein, it has a processor configured to search for coordinates corresponding to an object parallax that satisfies a specified condition along a second direction of a parallax map and update the coordinates to first coordinates, and calculate the height of an object corresponding to the object parallax based on the first coordinates. The parallax map is a parallax map in which a two-dimensional coordinate composed of a first direction corresponding to the horizontal direction of a captured image generated by a stereo camera capturing a road surface and a second direction intersecting the first direction corresponds to a parallax obtained from the captured image. The processor is configured to, when there are candidate coordinates corresponding to a parallax approximately equal to the object parallax at a position exceeding a specified interval from the first coordinates, calculate a first candidate height based on the first coordinates and calculate a second candidate height based on the candidate coordinates. Based on the parallaxes corresponding to two coordinates sandwiching the candidate coordinates in the first direction of the parallax map, determine which of the first candidate height and the second candidate height is to be taken as the height of the object. Parallaxes approximately equal to the object parallax include the same parallax as the object parallax and substantially the same parallax as the object parallax. A parallax with a difference from the object parallax within the range of ±10% is regarded as a parallax approximately equal to the object parallax.

2. The object detection device according to claim 1, wherein, the processor is configured to, when there are second coordinates corresponding to a parallax approximately equal to the object parallax within the specified interval from the first coordinates, update the second coordinates to the first coordinates.

3. The object detection device according to claim 2, wherein, the processor is configured to, search for coordinates corresponding to the object parallax from the coordinates of the road surface corresponding to the object parallax along a direction from the road surface upward among the directions included in the second direction of the parallax map, update them to first coordinates, and calculate the height of the object by subtracting the coordinates of the road surface from the first coordinates. Calculate the first candidate height by subtracting the coordinates of the road surface from the first coordinates, and calculate the second candidate height by subtracting the coordinates of the road surface from the candidate coordinates.

4. The object detection device according to claim 3, wherein, the processor is configured to, when there are no coordinates corresponding to a parallax approximately equal to the object parallax within a specified range along a direction from the coordinates of the road surface upward, not calculate the height of the object.

5. The object detection device according to claim 4, wherein, the specified range is set based on the height of an object floating from the road surface from the road surface.

6. The object detection device according to any one of claims 1 to 5, wherein, the processor is configured to, divide the parallax map into a plurality of partial regions along the first direction of the parallax map, and generate a distribution representing the frequency of parallax for each of the plurality of partial regions. Extract the parallax whose frequency of extraction exceeds a specified threshold as the target parallax that satisfies the specified condition.

7. The object detection device according to any one of claims 1 to 5, wherein the processor is configured to when it is determined that the parallaxes corresponding to two coordinates sandwiching the candidate coordinate are substantially equal to the target parallax, determine that the second candidate height among the first candidate height and the second candidate height is acquired as the height of the object. when it is not determined that the parallaxes corresponding to two coordinates sandwiching the candidate coordinate are substantially equal to the target parallax, determine that the first candidate height among the first candidate height and the second candidate height is acquired as the height of the object.

8. The object detection device according to any one of claims 1 to 5, wherein the specified interval is set based on the height of the object to be detected by the object detection device.

9. The object detection device according to any one of claims 1 to 5, wherein the processor executes the following processes: A first process of estimating the shape of the road surface in the actual space based on a first parallax map, the first parallax map being a map generated based on the output of the stereo camera and also a map in which a two-dimensional coordinate composed of the first direction and the second direction is associated with the parallax obtained from the output of the stereo camera; and A second process of generating a second parallax map obtained by removing, from the first parallax map, the parallax corresponding to a range at a height of a specified height or less from the road surface in the actual space as the parallax map.

10. An object detection system, wherein it has: a stereo camera that captures a plurality of images having parallax with each other; and an object detection device including at least one processor, the processor is configured to search for a coordinate corresponding to a target parallax that satisfies a specified condition along the second direction of the parallax map and update the coordinate to a first coordinate, and calculate the height of the object corresponding to the target parallax based on the first coordinate, the parallax map being a parallax map in which a two-dimensional coordinate composed of a first direction corresponding to the horizontal direction of a captured image generated by the stereo camera capturing a road surface and a second direction intersecting the first direction is associated with the parallax obtained from the captured image, when there is a candidate coordinate corresponding to a parallax substantially equal to the target parallax at a position exceeding a specified interval from the first coordinate, calculate a first candidate height based on the first coordinate and calculate a second candidate height based on the candidate coordinate, determine which one of the first candidate height and the second candidate height is acquired as the height of the object based on the parallaxes respectively corresponding to two coordinates sandwiching the candidate coordinate in the first direction of the parallax map, the parallax substantially equal to the target parallax includes a parallax identical to the target parallax and a parallax substantially identical to the target parallax, and a parallax whose difference from the target parallax is within the range of ±10% is regarded as a parallax substantially equal to the target parallax.

11. A moving body, wherein, There is an object detection system, which has: a stereo camera that captures a plurality of images having parallax with each other; and an object detection device including at least one processor, The processor is configured to, Search for coordinates corresponding to an object parallax that satisfies a specified condition along a second direction of the parallax map and update the coordinates to first coordinates, and calculate the height of an object corresponding to the object parallax based on the first coordinates. The parallax map is a parallax map in which a two-dimensional coordinate system composed of a first direction corresponding to the horizontal direction of a captured image generated by the stereo camera capturing a road surface and a second direction intersecting the first direction corresponds to the parallax obtained from the captured image, When there are candidate coordinates corresponding to a parallax that is approximately equal to the object parallax at a distance exceeding a specified interval from the first coordinates, calculate a first candidate height based on the first coordinates and calculate a second candidate height based on the candidate coordinates, Based on the parallaxes respectively corresponding to two coordinates sandwiching the candidate coordinates in the first direction of the parallax map, determine which one of the first candidate height and the second candidate height is to be obtained as the height of the object, The parallax that is approximately equal to the object parallax includes the same parallax as the object parallax and a parallax that is substantially the same as the object parallax. A parallax with a difference from the object parallax within the range of ±10% is regarded as a parallax that is approximately equal to the object parallax.

12. An object detection method, wherein, It includes the following steps: Search for coordinates corresponding to an object parallax that satisfies a specified condition along a second direction of the parallax map and update the coordinates to first coordinates, and calculate the height of an object corresponding to the object parallax based on the first coordinates. The parallax map is a parallax map in which a two-dimensional coordinate system composed of a first direction corresponding to the horizontal direction of a captured image generated by the stereo camera capturing a road surface and a second direction intersecting the first direction corresponds to the parallax obtained from the captured image, The step of calculating the height of the object includes: When there are candidate coordinates corresponding to a parallax that is approximately equal to the object parallax at a distance exceeding a specified interval from the first coordinates, calculate a first candidate height based on the first coordinates and calculate a second candidate height based on the candidate coordinates, Based on the parallaxes respectively corresponding to two coordinates sandwiching the candidate coordinates in the first direction of the parallax map, determine which one of the first candidate height and the second candidate height is to be obtained as the height of the object, The parallax that is approximately equal to the object parallax includes the same parallax as the object parallax and a parallax that is substantially the same as the object parallax. A parallax with a difference from the object parallax within the range of ±10% is regarded as a parallax that is approximately equal to the object parallax.

Images