Object detection device, object detection system, moving object, and object detection method

Abstract

The processor of the object detection device is configured to search for a coordinate corresponding to a parallax substantially equal to the object parallax that satisfies a specified condition along a second direction of the disparity map, update the coordinate to the first coordinate, and calculate the height of the object corresponding to the object parallax based on the first coordinate. The disparity map is a map that associates two-dimensional coordinates consisting of a first direction and a second direction intersecting the first direction with the parallax acquired from the captured image. The processor is configured to update the second coordinate to the first coordinate if a second coordinate corresponding to a parallax substantially equal to the object parallax exists within a specified interval from the first coordinate. The processor is configured not to update the second coordinate to the first coordinate if more than a specified number of coordinates corresponding to a parallax less than the object parallax exist between the first and second coordinates.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of priority from Japanese Patent Application No. 2019-170901, filed in Japan on September 19, 2019, and the disclosure of that prior application is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to an object detection device, an object detection system, a mobile object, and an object detection method. Background Art

[0004] In recent years, object detection devices using stereo cameras have been installed in mobile vehicles such as automobiles. These object detection devices acquire multiple images from the stereo camera and detect objects that may become obstacles based on the acquired multiple images (for example, see Patent Document 1).

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 5-265547. Summary of the Invention

[0008] An object detection device according to one embodiment of the present invention includes a processor. The processor is configured to search for coordinates corresponding to a parallax substantially equal to an object parallax satisfying a predetermined condition along a second direction of a disparity map, update the coordinates to first coordinates, and calculate the height of the object corresponding to the object parallax based on the first coordinates. The disparity map is a map in which two-dimensional coordinates consisting of a first direction corresponding to the horizontal direction of a captured image generated by a stereo camera capturing a road surface and a second direction intersecting the first direction are associated with the parallax acquired from the captured image. The processor is configured to update the second coordinate to the first coordinate if a second coordinate corresponding to a parallax substantially equal to the object parallax exists within a predetermined interval from the first coordinate. The processor is configured not to update the second coordinate to the first coordinate if more than a predetermined number of coordinates corresponding to a parallax less than the object parallax exist between the first and second coordinates.

[0009] An object detection system according to one embodiment of the present invention includes: a stereo camera that captures multiple images having mutual parallax; and an object detection device including at least one processor. The processor is configured to search for a coordinate corresponding to a parallax substantially equal to an object parallax that satisfies a predetermined condition along a second direction of a disparity map, update the coordinate to a first coordinate, and calculate the height of the object corresponding to the object parallax based on the first coordinate. The disparity map is a map that associates two-dimensional coordinates consisting of a first direction corresponding to the horizontal direction of an image captured by the stereo camera capturing a road surface and a second direction intersecting the first direction with the parallax acquired from the captured image. The processor is configured to update the second coordinate to the first coordinate if a second coordinate corresponding to a parallax substantially equal to the object parallax exists within a predetermined interval from the first coordinate. The processor is configured not to update the second coordinate to the first coordinate if a predetermined number of coordinates corresponding to a parallax less than the object parallax exist between the first and second coordinates.

[0010] A mobile object according to one embodiment of the present invention includes an object detection system comprising: a stereo camera that captures multiple images having parallaxes with respect to each other; and an object detection device including at least one processor. The processor is configured to search for coordinates corresponding to a parallax substantially equal to an object parallax that satisfies a predetermined condition along a second direction of a disparity map, update the coordinates to first coordinates, and calculate the height of an object corresponding to the object parallax based on the first coordinates. The disparity map is a map that associates two-dimensional coordinates consisting of a first direction corresponding to the horizontal direction of an image captured by the stereo camera and a second direction intersecting the first direction with parallaxes acquired from the captured image. The processor is configured to update the second coordinate to the first coordinate if a second coordinate corresponding to a parallax substantially equal to the object parallax exists within a predetermined interval from the first coordinate. The processor is configured not to update the second coordinate to the first coordinate if a predetermined number of coordinates corresponding to parallaxes less than the object parallax exist between the first and second coordinates.

[0011] An object detection method according to one embodiment of the present invention includes the step of calculating the height of an object corresponding to an object parallax based on a first coordinate. Calculating the object's height includes searching for a coordinate corresponding to a parallax substantially equal to the object's parallax that satisfies a specified condition along a second direction of a disparity map and updating the coordinate to the first coordinate. The disparity map is a map that associates two-dimensional coordinates consisting of a first direction corresponding to the horizontal direction of an image captured by a stereo camera capturing a road surface and a second direction intersecting the first direction with the parallax acquired from the captured image. Calculating the object's height includes updating the second coordinate to the first coordinate if a second coordinate corresponding to a parallax substantially equal to the object's parallax exists within a specified interval from the first coordinate. Calculating the object's height includes not updating the second coordinate to the first coordinate if more than a specified number of coordinates corresponding to parallaxes less than the object's parallax exist between the first and second coordinates. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a block diagram showing a schematic configuration of an object detection system according to one embodiment of the present invention.

[0013] Figure 2 It schematically shows the Figure 1 A side view of a moving body of the object detection system is shown.

[0014] Figure 3 It is a schematic representation of the Figure 1 A front view of a moving body of the object detection system shown.

[0015] Figure 4 This is a block diagram showing a schematic configuration of an object detection system according to another embodiment of the present invention.

[0016] Figure 5 Yes Figure 1 Flowchart of an example of the flow of processing executed by the object detection device shown.

[0017] Figure 6 A diagram illustrating an example of a first parallax image acquired or generated by the object detection device.

[0018] Figure 7 This is a flowchart showing an example of a process for estimating the shape of a road surface.

[0019] Figure 8 This is a flowchart showing an example of a process of extracting road surface candidate parallax from a first parallax image.

[0020] Figure 9A diagram illustrating the positional relationship between the road surface and the stereo camera.

[0021] Figure 10 This is a diagram explaining the procedure for extracting the parallax of road surface candidates.

[0022] Figure 11 This is a diagram showing the range on the road surface where parallax is histogrammed.

[0023] Figure 12 is the road parallax d r A dv correlation diagram showing an example of the relationship between the vertical coordinate (v coordinate).

[0024] Figure 13 This is a diagram explaining a method of detecting whether an object other than road surface parallax is included.

[0025] Figure 14 It uses a straight line to approximate the road parallax d r Flowchart of processing of the relationship between the vertical coordinate (v coordinate) of the image.

[0026] Figure 15 This is to illustrate the use of the first straight line to approximate the road parallax d r Picture.

[0027] Figure 16 This is a diagram explaining the method of determining the second straight line.

[0028] Figure 17 It means using a straight line to approximate the road parallax d r A diagram showing an example of the result of the relationship between the vertical coordinate (v coordinate) of the image.

[0029] Figure 18 A diagram showing an example of a second parallax image.

[0030] Figure 19 It is the reference image of the second parallax image.

[0031] Figure 20 This is a flowchart showing an example of the detection process of the first parallax and the second parallax.

[0032] Figure 21 is Figure 18 A partial area is shown superimposed on the second parallax image.

[0033] Figure 22 This is a diagram showing an example of a parallax histogram.

[0034] Figure 23 This is a flowchart (part 1) showing an example of a calculation process for calculating the height of an object.

[0035] Figure 24 This is a flowchart (part 2) showing an example of a calculation process for calculating the height of an object.

[0036] Figure 25 This is a flowchart (part 3) showing an example of a calculation process for calculating the height of an object.

[0037] Figure 26 A diagram showing an example of a second parallax image.

[0038] Figure 27 A diagram showing an example of a second parallax image.

[0039] Figure 28 is with Figure 27 The second parallax image shown corresponds to the first image.

[0040] Figure 29 A diagram showing an example of a second parallax image.

[0041] Figure 30 is with Figure 29 The second parallax image shown corresponds to the first image.

[0042] Figure 31 This is a flowchart showing an example of parallel object detection processing.

[0043] Figure 32 This is a diagram showing an example of a UD diagram.

[0044] Figure 33 is with Figure 32 The UD diagram shown corresponds to the first image.

[0045] Figure 34 This is a diagram showing a UD diagram excluding a point group substantially parallel to the u direction.

[0046] Figure 35 This is a diagram (part 1) explaining the Hough transform.

[0047] Figure 36 This is a diagram explaining the Hough transform (part 2).

[0048] Figure 37 This is a flowchart showing an example of restoration processing.

[0049] Figure 38 This is a diagram showing an example of a UD diagram.

[0050] Figure 39 This is a diagram showing an example of a UD map to which a restoration flag is added.

[0051] Figure 40 A diagram showing an example of a second parallax image.

[0052] Figure 41 A diagram showing an example of a second parallax image.

[0053] Figure 42 Is to express Figure 41 The second parallax image shown corresponds to the first image.

[0054] Figure 43 It means in Figure 40 FIG. 1 shows a diagram of parallax pixels used to determine the height of an object in a second parallax image.

[0055] Figure 44 It means in Figure 41 FIG. 1 shows a diagram of parallax pixels used to determine the height of an object in a second parallax image.

[0056] Figure 45 This is a flowchart showing an example of a process of determining a representative parallax.

[0057] Figure 46 An example of the acquired representative parallax corresponding to the u direction is shown.

[0058] Figure 47 An example of the averaged representative parallax corresponding to the u direction is shown.

[0059] Figure 48 This is a diagram showing an example of distribution in a UD map showing a point group representing parallax.

[0060] Figure 49 This is a diagram of the road surface viewed from the height direction (y direction).

[0061] Figure 50 This is a diagram transformed into a point group on the xz plane representing the real space of parallax.

[0062] Figure 51 This is a diagram showing an example of a method for outputting object detection results. DETAILED DESCRIPTION

[0063] In existing object detection devices, it is necessary to improve the performance of detecting objects. The object detection device, object detection system, mobile object and object detection method of the present invention can improve the performance of detecting objects.

[0064] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. In the following drawings, the same or similar components are given the same figure marks. In addition, the figures used in the following description are schematic figures. The dimensions and proportions on the drawings may not necessarily be consistent with the objects in reality. The figures showing the captured images and parallax images captured by the camera include figures created for illustration. These images are different from the images actually captured or processed. In addition, in the following description, the "subject" is an object captured by the camera. The "subject" includes objects, roads, and the sky, etc. An "object" is an object with a specific position and size in space. An "object" is also called a "three-dimensional object."

[0065] like Figure 1 As shown, the object detection system 1 includes a stereo camera 10 and an object detection device 20. The stereo camera 10 and the object detection device 20 can communicate through wired or wireless communication. The stereo camera 10 and the object detection device 20 can communicate through a network. The network may include, for example, a wired or wireless LAN (Local Area Network), or a CAN (Controller Area Network). The stereo camera 10 and the object detection device 20 can be housed in the same housing and configured as a whole. The stereo camera 10 and the object detection device 20 can be configured to be located in a moving body 30 described later, and can communicate with an ECU (Electronic Control Unit) in the moving body 30.

[0066] In the present invention, a "stereo camera" is a plurality of cameras that have parallax and cooperate with each other. A stereo camera includes at least two or more cameras. In a stereo camera, multiple cameras can cooperate to shoot an object from multiple directions. A stereo camera can be a device that includes multiple cameras in one housing. A stereo camera can be a device that includes two or more cameras that are independent of each other and separated from each other. A stereo camera is not limited to multiple independent cameras. In the present invention, for example, a camera having an optical mechanism that guides light incident on two separated parts to a light receiving element can be used as a stereo camera. In the present invention, multiple images of the same subject taken from different viewpoints are sometimes referred to as "stereo images."

[0067] like Figure 1As shown, the stereo camera 10 has a first camera 11 and a second camera 12. The first camera 11 and the second camera 12 respectively have an optical system and a shooting element that define the optical axis OX. The first camera 11 and the second camera 12 respectively have different optical axes OX. In this embodiment, only a single figure mark OX is used to collectively represent the optical axes OX of the first camera 11 and the second camera 12. The shooting element includes a CCD image sensor (Charge-Coupled Device Image Sensor) and a CMOS image sensor (Complementary MOS Image Sensor). The shooting elements respectively possessed by the first camera 11 and the second camera 12 can be located in the same plane perpendicular to the optical axis OX of each camera. The first camera 11 and the second camera 12 generate an image signal representing the image formed by the shooting element. In addition, the first camera 11 and the second camera 12 can perform arbitrary processing such as distortion correction, brightness adjustment, contrast adjustment and gamma correction on the captured image.

[0068] The optical axes OX of the first camera 11 and the second camera 12 are oriented in directions such that they can both capture the same subject. The optical axes OX and positions of the first camera 11 and the second camera 12 are determined so that the captured images contain at least the same subject. The optical axes OX of the first camera 11 and the second camera 12 are oriented parallel to each other. This parallelism is not strictly parallel; assembly deviations, installation deviations, and these deviations over time are allowed. The optical axes OX of the first camera 11 and the second camera 12 are not necessarily parallel and can be oriented in different directions. Even when the optical axes OX of the first camera 11 and the second camera 12 are not parallel to each other, a stereoscopic image can be generated by transforming the images within the stereo camera 10 or the object detection device 20. The baseline length is the distance between the optical center of the first camera 11 and the optical center of the second camera 12. The baseline length is equivalent to the distance between the centers of the lenses of the first camera 11 and the second camera 12. The baseline length direction is the direction connecting the optical center of the first camera 11 and the optical center of the second camera 12.

[0069] The first camera 11 and the second camera 12 are arranged separately in a direction intersecting the optical axis OX. In one embodiment of multiple embodiments, the first camera 11 and the second camera 12 are arranged along the left-right direction. The first camera 11 is located on the left side of the second camera 12 when facing forward. The second camera 12 is located on the right side of the first camera 11 when facing forward. The first camera 11 and the second camera 12 shoot the subject at a specified frame rate (Frame rate) (for example, 30fps). Due to the difference in the positions of the first camera 11 and the second camera 12, the positions of the corresponding subjects in the two images shot by each camera are different. The first camera 11 shoots the first image. The second camera 12 shoots the second image. The first image and the second image are stereo images shot from different viewpoints.

[0070] like Figure 2 as well as Figure 3 As shown, the object detection system 1 is mounted on a moving object 30. Figure 2 As shown, the first camera 11 and the second camera 12 are arranged so as to be able to capture the front of the moving object 30 so that the optical axis OX of each optical system of the first camera 11 and the second camera 12 is substantially parallel to the front of the moving object 30 .

[0071] The mobile body 30 of the present invention travels on a travel path including a road or a track, etc. The surface of the travel path on which the mobile body 30 travels is also referred to as a "road surface."

[0072] In the present invention, the direction of movement of the moving object 30 when moving straight ahead is also referred to as "forward" or the "positive z-axis direction." The direction opposite to the forward direction is also referred to as "rearward" or the "negative z-axis direction." Unless otherwise specified, the positive and negative z-axis directions are collectively referred to as the "z-direction." Left and right directions are defined with respect to the forward-facing state of the moving object 30. The z-direction is also referred to as the "depth direction."

[0073] In the present invention, the direction perpendicular to the z-direction and extending from the left to the right is also referred to as the "positive x-axis direction." The direction perpendicular to the z-direction and extending from the right to the left is also referred to as the "negative x-axis direction." Unless otherwise specified, the positive and negative x-axis directions are collectively referred to as the "x-direction." The x-direction may coincide with the baseline length direction. The x-direction is also referred to as the "horizontal direction."

[0074] In the present invention, the direction perpendicular to the road surface near the moving object 30 and pointing upward from the road surface is also referred to as the "height direction" or the "positive y-axis direction." The direction opposite to the height direction is also referred to as the "negative y-axis direction." Unless otherwise specified, the positive and negative y-axis directions are collectively referred to as the "y-direction." The y-direction may be orthogonal to the x-direction and the z-direction. The y-direction is also referred to as the "vertical direction."

[0075] The "mobile body" in the present invention may include, for example, vehicles and aircraft. Vehicles may include, for example, automobiles, industrial vehicles, railway vehicles, household vehicles, and fixed-wing aircraft that travel on runways. Automobiles may include, for example, cars, trucks, buses, two-wheeled vehicles, and trolleybuses. Industrial vehicles may include, for example, industrial vehicles used in agriculture and construction. Industrial vehicles may include, for example, forklifts and golf carts. Industrial vehicles used in agriculture may include, for example, tractors, tillers, transplanters, binders, combine harvesters, and mowers. Industrial vehicles used in construction may include, for example, bulldozers, scrapers, excavators, cranes, dump trucks, and loading and unloading trucks. Vehicles may include vehicles that are driven by human power. The classification of vehicles is not limited to the above examples. For example, automobiles may include industrial vehicles that can travel on roads. The same vehicle may be included in multiple classifications. Aircraft may include, for example, fixed-wing aircraft and rotary-wing aircraft.

[0076] The first camera 11 and the second camera 12 can be mounted at various locations on the mobile object 30. In one embodiment, the first camera 11 and the second camera 12 can be mounted inside the mobile object 30, which is a vehicle, and can capture images of the exterior of the mobile object 30 through the windshield. For example, the first camera 11 and the second camera 12 are positioned in front of the rearview mirror or on the dashboard. In one embodiment, the first camera 11 and the second camera 12 can be mounted on any of the vehicle's front bumper, fender grille, side fenders, light module, and hood.

[0077] The object detection device 20 can be located anywhere within the mobile object 30. For example, the object detection device 20 can be located within the instrument panel of the mobile object 30. The object detection device 20 acquires a first image and a second image from the stereo camera 10. The object detection device 20 detects an object based on the first and second images. If the mobile object 30 is a vehicle, the object to be detected by the object detection device 20 can be an object on the road surface. Examples of such objects on the road surface include other vehicles and pedestrians.

[0078] The object detection device 20 can be configured to read a program recorded in a non-temporary computer-readable medium to implement the processing performed by the control unit 24 described below. Non-temporary computer-readable media include, but are not limited to, magnetic storage media, optical storage media, magneto-optical storage media, and semiconductor storage media. Magnetic storage media include magnetic disks, hard disks, and magnetic tapes. Optical storage media include CDs (Compact Discs), DVDs, Blu-ray Discs (Blu-ray (registered trademark) Discs), and other optical disks. Semiconductor storage media include ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), and flash memory.

[0079] The object detection device 20 includes an acquisition unit 21 , an output unit 22 , a memory 23 , and a control unit 24 (processor).

[0080] The acquisition unit 21 is an input interface of the object detection device 20. The acquisition unit 21 can receive information input from the stereo camera 10 and other devices. The acquisition unit 21 can use physical connectors and wireless communication devices. Physical connectors include: electrical connectors corresponding to transmission using electrical signals, optical connectors corresponding to transmission using optical signals, and electromagnetic connectors corresponding to transmission using electromagnetic waves. Electrical connectors include: connectors compliant with IEC60603, connectors compliant with USB standards, connectors corresponding to RCA terminals, connectors corresponding to S terminals specified in EIAJ CP-1211A, connectors corresponding to D terminals specified in EIAJ RC-5237, connectors compliant with HDMI (registered trademark) standards, and connectors corresponding to coaxial cables including BNC. Optical connectors include: various connectors compliant with IEC 61754. Wireless communication devices include: Bluetooth (registered trademark) and wireless communication devices compliant with various standards including IEEE802.11. The wireless communication device includes at least one antenna.

[0081] The acquisition unit 21 may input image data of images captured by the first camera 11 and the second camera 12. The acquisition unit 21 outputs the input image data to the control unit 24. The acquisition unit 21 may be compatible with the transmission method of the imaging signal of the stereo camera 10. The acquisition unit 21 may be connected to the output interface of the stereo camera 10 via a network.

[0082] The output unit 22 is an output interface of the object detection device 20. The output unit 22 can output the processing results of the object detection device 20 to other devices within the mobile body 30 or other devices outside the mobile body 30. Other devices within the mobile body 30 may include driving assistance devices such as automatic cruise control and safety devices such as automatic braking devices. Other devices outside the mobile body 30 may include other vehicles and road detectors. Other devices within the mobile body 30 or other devices outside the mobile body 30 can appropriately use the information received from the object detection device 20. The output unit 22 can be the same as or similar to the acquisition unit 21 and can include various interfaces corresponding to wired and wireless communications. For example, the output unit 22 can have a CAN interface. In this case, the output unit 22 communicates with other devices within the mobile body 30.

[0083] The memory 23 stores programs used for various processes and information used in calculations. The memory 23 includes volatile memory and non-volatile memory. The memory 23 includes memory independent of the processor and memory built into the processor.

[0084] The control unit 24 includes one or more processors. Processors include general-purpose processors that read specific programs and execute specific functions, and dedicated processors dedicated to specific processing. Dedicated processors include application-specific integrated circuits (ASICs). Processors include programmable logic devices (PLDs). PLDs include FPGAs (Field-Programmable Gate Arrays). The control unit 24 can be any of an SoC (System-on-a-Chip) and a SiP (System In a Package) in which one or more processors collaborate. The processing performed by the control unit 24 can be said to be the processing performed by the processor.

[0085] The control unit 24 performs various processes on the first disparity map during information processing within the object detection device 20. The first disparity map is a map that associates two-dimensional coordinates with disparities. The two-dimensional coordinates of the first disparity map are composed of a horizontal direction corresponding to the horizontal direction of the image captured by the stereo camera 10 and a vertical direction intersecting the horizontal direction. The horizontal direction is a first direction. The vertical direction is a second direction. The horizontal and vertical directions may be orthogonal to each other. The horizontal direction may correspond to the width direction of the road surface. When the image captured by the stereo camera 10 includes a horizontal line, the horizontal direction may correspond to a direction parallel to the horizontal line. The vertical direction may correspond to the direction in which gravity is applied in real space.

[0086] During information processing within the object detection device 20, the first parallax map undergoes various operations. These operations include computational processing, writing to and reading from the memory 23, and the like. The image obtained by visualizing the first parallax map is also referred to as a "first parallax image." A first parallax image is an image in which pixels representing parallax are arranged on a two-dimensional plane formed by horizontal and vertical directions. The following describes how the control unit 24 performs various operations on the first parallax image. In the following description, processing of the first parallax image can be referred to as processing of the first parallax map.

[0087] The structure of the object detection system 1 of the present invention is not limited to Figure 1 The structure shown. Figure 4 An object detection system 1A according to another embodiment of the present invention is shown. The object detection system 1A includes a stereo camera 10, an object detection device 20, and a generation device 25. The generation device 25 can be mounted on hardware separate from the object detection device 20. The generation device 25 generates a first parallax image based on the first image and the second image output from the stereo camera 10. The generation device 25 includes a processor. The processor included in the generation device 25 generates the first parallax image based on the first image and the second image output from the first camera 11 and the second camera 12 of the stereo camera 10, respectively. The acquisition unit 21 acquires the first parallax image from the generation device 25. The object detection device 20 and the generation device 25 can be collectively regarded as one object detection device.

[0088] Below, refer to Figure 5 The flowchart shown in FIG. 1 illustrates the processing executed by the control unit 24 . Figure 5 Yes Figure 1 Flowchart showing an example of the overall flow of processing executed by the object detection device 20 shown.

[0089] First, in detail Figure 5 Before describing the processing performed in each step of the flowchart, an overview and purpose of the processing in each step will be briefly described.

[0090] Step S101 is a step of acquiring or generating a first parallax image. Step S101 is equivalent to the first processing described later. Figure 1 In the structure shown, the control unit 24 generates a first parallax image. Figure 4 In the illustrated configuration, the control unit 24 acquires the first parallax image generated by the generating device 25 via the acquiring unit 21 .

[0091] Step S102 is the step of estimating the shape of the road surface. The process performed in step S102 is also referred to as the "first process." By estimating the road surface shape, parallax corresponding to the road surface relative to the longitudinal coordinates can be estimated in the first parallax image. The road surface shape can be used to remove unnecessary parallax in the subsequent processes and / or to estimate the height position of the road surface in real space.

[0092] Step S103 is a step of generating a second parallax image by removing unnecessary parallax from the first parallax image. The processing performed in step S103 is also referred to as "second processing." Unnecessary parallax includes parallax corresponding to white lines on the road surface and parallax corresponding to buildings above the road surface, which may be included in the first parallax image. By removing unnecessary parallax from the first parallax image, the object detection device 20 reduces the possibility of mistakenly detecting white lines on the road surface and buildings above the road surface as objects to be detected on the road surface. This can improve the accuracy of object detection by the object detection device 20.

[0093] Step S103 is a step of generating a second disparity map by removing unnecessary disparity from the first disparity map during information processing within object detection device 20. The second disparity image is an image obtained by converting the second disparity map into an image. In the following description, processing the second disparity image may also be referred to as processing the second disparity map.

[0094] Step S104 is a step of detecting the first and second parallaxes based on the second parallax image. The first parallax is detected by considering it as the parallax corresponding to the object being detected. The second parallax is detected by considering it as a candidate for the parallax corresponding to the object being detected. Whether the second parallax should be restored as the parallax corresponding to the object being detected in the restoration process described later can be determined.

[0095] Step S105 is a step of calculating the height of the object on the image. The height of the object on the image detected can be as described below. Figure 51 The height of the detection frame 182 shown in FIG. 18 is also referred to as the height of the object on the image. The processing executed in step S105 is also referred to as the "height calculation processing."

[0096] Step S106 is a step of detecting the parallax corresponding to an object parallel to the direction of travel of the mobile body 30 in the second parallax image. An object parallel to the direction of travel of the mobile body 30 is also referred to as a "parallel object". Examples of parallel objects include guardrails, roadside buildings such as soundproof walls on highways, or the sides of other vehicles. For example, in the second parallax image, parallel objects such as guardrails and detection target objects such as other vehicles may be close. By detecting parallel objects, for example, by adding a mark to the detected parallel objects, the possibility of the object detection device 20 mistakenly detecting parallel objects as detection target objects on the road surface is reduced. With such a structure, the accuracy of the object detection device 20 in detecting objects can be improved. The processing performed in step S106 is also referred to as "parallel object detection processing."

[0097] Step S107 is a step of determining whether or not the second parallax is restored to the parallax corresponding to the object to be detected.

[0098] Step S108 is a step for determining a representative parallax from the first parallax and the restored second parallax at each horizontal coordinate in the second parallax image. The processing performed in step S108 is also referred to as the "third processing." The processing performed in steps S104 to S108 is also referred to as the "third processing."

[0099] Step S109 is a step for detecting an object by converting information representing parallax into coordinates in real space and extracting a set (group) representing the parallax. The process performed in step S109 is also referred to as the "fourth process." In step S109, the position of the object to be detected and the width of the object as viewed from the stereo camera 10 are obtained.

[0100] Step S110 is a step of outputting information about the detected object from the output unit 22. The object information output in step S110 may include the height of the object on the image calculated in step S105, the position of the object detected in step S109, and the width of the object as viewed from the stereo camera 10 side detected in step S109. This information may be provided to other devices within the mobile object 30.

[0101] Next, each step is described in detail.

[0102] In the process of step S101, the control unit 24 acquires or generates a first parallax image. Figure 1 In the object detection system 1 shown in FIG. 1 , the control unit 24 generates a first parallax image based on the first image and the second image acquired by the acquisition unit 21 . Figure 4In the object detection system 1A shown, the control unit 24 acquires the first parallax image generated by the generation device 25 via the acquisition unit 21. The control unit 24 may store the first parallax image in the memory 23 for subsequent processing.

[0103] The method of generating the first parallax image is well known, so it will be briefly described below. In the following, it is assumed that the control unit 24 generates the first parallax image.

[0104] The control unit 24 obtains a first image captured by the first camera 11 and a second image captured by the second camera 12. The control unit 24 divides one of the first and second images (for example, the first image) into multiple small areas. A small area can be a rectangular area with multiple pixels arranged vertically and horizontally. For example, a small area can be composed of three pixels arranged vertically and three pixels arranged horizontally. However, the number of pixels arranged vertically and horizontally in a small area is not limited to three. In addition, the number of pixels in the vertical and horizontal directions of a small area can be different. The control unit 24 shifts each pixel in the divided small areas horizontally on a pixel-by-pixel basis on the other image and compares feature quantities to perform matching. For example, when the first image is divided into small areas, the control unit 24 shifts the small areas of the first image horizontally on a pixel-by-pixel basis on the second image and compares feature quantities to perform matching. Feature quantities include, for example, brightness and color patterns. A method using the SAD (Sum of Absolute Differences) function is known for matching stereo images. Here, it represents the sum of the absolute values of the differences in brightness values within a small area. When the SAD function is minimum, the two images are judged to be most similar. Stereo image matching is not limited to methods using the SAD function. Other methods can also be used in stereo image matching.

[0105] The control unit 24 calculates the disparity of each small area based on the difference in the horizontal position of the pixels of the two areas matched in the first image and the second image. The disparity can be the difference between the position of the same subject in the first image and the position in the second image. The size of the disparity can be expressed in units of the horizontal width of the pixels on the stereo image. By performing interpolation processing, the size of the disparity can be calculated with an accuracy of less than one pixel. The size of the disparity corresponds to the distance between the subject photographed by the stereo camera 10 and the stereo camera 10 in the actual space. The closer the distance from the stereo camera 10 to the subject in the actual space, the larger the disparity corresponding to the subject. The farther the distance from the stereo camera 10 to the subject in the actual space, the smaller the disparity corresponding to the subject.

[0106] The control unit 24 generates a first parallax image representing the distribution of the calculated parallax. Pixels representing the parallax that constitute the first parallax image are also called "parallax pixels." The control unit 24 can generate the first parallax image with the same resolution as the pixels of the original first and second images.

[0107] Figure 6 . The first parallax image 40 is shown in the figure. The first parallax image 40 is a two-dimensional plane composed of the horizontal direction (first direction) of the stereo camera 10 and the vertical direction (second direction) orthogonal to the horizontal direction. Parallax pixels representing parallax are arranged in the two-dimensional plane of the first parallax image 40. The horizontal direction is also called the "u direction". The vertical direction is also called the "v direction". The coordinate system composed of the u direction and the v direction is also called the "uv coordinate system" and the "image coordinate system". In this embodiment, the corner of each figure facing the upper left side of the paper is the origin (0, 0) of the uv coordinate system. In addition, the direction from the left side to the right side of the paper of each figure is the positive direction of the u axis, and the opposite direction is the negative direction of the u axis. In addition, the direction from the top to the bottom of the paper of each figure is the positive direction of the v axis, and the opposite direction is the negative direction of the v axis. The negative direction of the v axis corresponds to the direction from the road surface upward in real space. The u coordinate and the v coordinate can be expressed in units of parallax pixels.

[0108] like Figure 6 As shown, the first parallax image 40 includes parallax image 41, parallax image 42, and parallax image 43. Parallax image 41 corresponds to the road surface in front of the moving object 30. Parallax image 42 corresponds to another vehicle in front of the moving object 30. Parallax image 43 is a parallax image corresponding to a guardrail.

[0109] The control unit 24 can display the parallax information of each pixel of the first parallax image by the brightness or color of each pixel. Figure 6 In the first parallax image 40 shown, the parallax of each pixel is displayed by different shades for ease of explanation. In the first parallax image 40, the darker the shade, the smaller the parallax represented by the pixels in the area. The thinner the shade, the larger the parallax represented by the pixels in the area. In the first parallax image 40, the pixels in the area with equal shades all represent parallax within a specified range. In an actual first parallax image, since the pixels in a part of the area have fewer feature quantities on the stereo image in the above-mentioned matching process for calculating the parallax than the pixels in other areas, it is sometimes difficult to calculate the parallax. For example, it is difficult to calculate the parallax for parts of a spatially uniform subject such as the windows of a vehicle, and for parts where highlight overflow (whitening) caused by reflection of sunlight occurs. In the first parallax image, when there is parallax corresponding to objects and buildings, they can be represented with a brightness or color different from the parallax corresponding to the background located farther away.

[0110] The control unit 24 may not display the first parallax image as an image after calculating the parallax. In other words, the control unit 24 only needs to retain the first parallax map that is the basis of the first parallax image and perform appropriate processing on the first parallax map.

[0111] After executing step S101, the control unit 24 executes a first process of estimating the shape of the road surface based on the first parallax image (step S102). Figure 7 、 Figure 8 as well as Figure 14 The process of estimating the shape of the road surface performed by the control unit 24 will be described with reference to the flowchart of FIG. First, the control unit 24 extracts the road surface candidate parallax d from the first parallax image. c (Step S201) Road surface candidate parallax d c is the road disparity d collected from the first disparity image r Parallax with a high probability of matching. Road parallax d r Refers to the parallax of the road area. r The parallax of objects on the road is not included. r Indicates the distance to the corresponding point on the road surface. Road surface parallax d r Values having close values at positions having the same v coordinate are collected.

[0112] exist Figure 8 The flowchart shows the road candidate parallax d c The specific content of the extraction process. Figure 8 As shown, the control unit 24 calculates the initial value of parallax used to calculate the road candidate parallax, namely, the road candidate parallax initial value d0, based on the installation position of the stereo camera 10 (step S301). The road candidate parallax initial value d0 is the initial value of the road candidate parallax at the road candidate parallax extraction position closest to the stereo camera 10. The road candidate parallax extraction position closest to the stereo camera 10 can be set, for example, within a range of 1 meter to 10 meters from the stereo camera.

[0113] like Figure 9As shown, in the stereo camera 10, the road surface height Y is the height of the stereo camera 10 in the vertical direction from the photographed road surface 41A. In addition, the road surface height Y0 is the height of the installation position of the stereo camera 10 from the road surface 41A. Due to the undulations of the road, the road surface height Y may change depending on the distance from the stereo camera 10. Therefore, the road surface height Y at a position far from the stereo camera 10 is inconsistent with the road surface height Y0 at the installation position of the stereo camera 10. In one embodiment of the multiple embodiments, it is assumed that the first camera 11 and the second camera 12 of the stereo camera 10 are arranged in such a manner that the optical axes OX are parallel to each other and face forward. Figure 9 In the equation, the distance Z represents the horizontal distance to a specific road surface position. Let the baseline length of the stereo camera 10 be B, and the vertical image size be TOTALv. In this case, the road parallax d of the road surface 41A captured at a certain vertical coordinate (v coordinate) is s The relationship between the height of the road surface and Y is given by the following mathematical formula.

[0114] d s =B / Y×(v-TOTALv / 2) (1)

[0115] The road parallax d calculated by mathematical formula (1) is s It is also called "geometrically estimated road surface parallax". s Indicates the geometrically estimated road surface parallax.

[0116] The road surface candidate parallax initial value d0 is the road surface candidate parallax d between the assumed road surface 41A at the position between the stereo camera 10 and the position closest to the stereo camera 10. c The value calculated when the extraction positions are parallel to and flat with the optical axis OX of the first camera 11 and the second camera 12. In this case, the v coordinate of the extraction position of the road candidate parallax at the position closest to the stereo camera 10 is determined as a specific coordinate (v0) on the first parallax image. The coordinate (v0) is the initial value of the v coordinate for extracting the road candidate parallax. The coordinate (v0) is between TOTALv / 2 and TOTALv. The coordinate (v0) is located at the bottom (the larger side of the v coordinate) within the range of image coordinates capable of calculating the parallax. The coordinate (v0) can be TOTALv corresponding to the bottom row of the first parallax image. The road candidate parallax initial value d0 can be determined by substituting v0 for v in mathematical formula (1) and substituting Y0 for Y.

[0117] Based on the initial road surface candidate parallax value d0, the control unit 24 calculates the parallax collection threshold for the first row, whose vertical v coordinate is the coordinate (v0) (step S302). A row refers to an array of pixels arranged horizontally with the same v coordinate on the first parallax image. The parallax collection threshold includes an upper threshold, which is the upper limit threshold for collecting parallax, and a lower threshold, which is the lower limit threshold for collecting parallax. Based on a prescribed rule, the parallax collection threshold is set above and below the initial road surface candidate parallax value d0 so that the parallax collection threshold includes the initial road surface candidate parallax value d0. Specifically, the road surface parallax when the road surface height Y changes by a predetermined road surface height change ΔY from the state where the road surface candidate parallax initial value d0 was calculated is determined as the upper and lower thresholds of the parallax collection threshold. That is, the lower threshold of the parallax collection threshold is obtained by subtracting the parallax corresponding to the road surface height change ΔY from the initial road surface candidate parallax value d0. The upper limit of the parallax collection threshold is obtained by adding the parallax of the road surface height change ΔY to the initial parallax value d0 of the road surface candidate. The lower limit and upper limit of the parallax collection threshold are specifically obtained by changing the value of Y in equation (1).

[0118] Thereafter, the control unit 24 repeatedly executes the processing from step S303 to step S307. First, the control unit 24 processes a row whose v coordinate is the coordinate (v0) located at the bottom of the first parallax image (step S303).

[0119] The control unit 24 collects disparities using the disparity collection threshold (step S304). The control unit 24 collects disparity pixels having a disparity between the lower threshold and the upper threshold of the disparity collection threshold as road surface candidate disparities d c That is, the control unit 24 determines that the parallax pixel having the parallax within the predetermined limit range based on the road surface candidate parallax initial value d0 calculated using the mathematical formula (1) is a candidate for the parallax pixel representing the correct parallax of the road surface 41A. The control unit 24 sets the parallax of the parallax pixel determined as a candidate for the parallax pixel representing the correct parallax of the road surface 41A as the road surface candidate parallax d c With this configuration, the control unit 24 can reduce the possibility of misjudging parallax corresponding to objects other than the road surface 41A, such as objects on the road surface 41A and buildings, as parallax corresponding to the road surface 41A. This improves the detection accuracy of the road surface 41A.

[0120] In the process of step S304, when the determination of all the parallax pixels whose v coordinate is the coordinate (v0) is completed, the control unit 24 sets the collected road surface candidate parallax d c Averaging is performed to calculate the road candidate parallax d cThe average value of the average road candidate parallax d av (Step S305). The control unit 24 can calculate the parallax d of each road candidate. c Its uv coordinates and v coordinates are the average road candidate parallax d of the coordinate (v0) av Stored in memory 23.

[0121] After executing the process of step S305, the control unit 24 executes the process of step S306. In the process of step S306, the control unit 24 calculates the average road surface candidate parallax d calculated in the process of step S305 for each parallax pixel on a row, that is, a row where the v coordinate is the coordinate (v0-1). av , calculates the parallax collection threshold. The control unit 24 calculates the average road surface candidate parallax d when the v coordinate calculated in the process of step S305 is the coordinate (v0). av , to change the road height Y so that the mathematical formula (1) is established. The control unit 24 substitutes v0-1 for v0 in the mathematical formula (1) with the changed road height Y, and calculates the geometrically estimated road parallax d when the v coordinate is the coordinate (v0-1). s The control unit 24 can estimate the road surface parallax d from the geometry in a manner similar to the process of step S302. s The control unit 24 can set the geometrically estimated road surface parallax d to be the lower limit threshold of the parallax collection threshold. s The parallax obtained by adding the parallax corresponding to the predetermined road surface height change amount ΔY is set as the upper limit threshold of the parallax collection threshold.

[0122] After executing the process of step S306, the control unit 24 determines the geometrically estimated road surface parallax d calculated by the equation (1). s Is it greater than a specified value. The specified value is, for example, 1 pixel. s If it is greater than 1, the control unit 24 returns to the process of step S303 (step S307). In the process of step S303, the control unit 24 sets the road surface candidate parallax d c The extracted object moves to a row above one pixel. That is, the road candidate disparity d c When the object to be extracted is in a row with v coordinate (v0), the control unit 24 changes the v coordinate of the row of the object detected on the road surface to coordinate (v0-1). Figure 10 As shown, the road candidate parallax d c When the object of calculation of is the nth row, the control unit 24 changes the row of the object of road surface detection to the n+1th row. Figure 10The vertical width of each row is enlarged. The actual height of each row is 1 pixel. In this case, the v coordinate of the n+1th row is 1 less than the v coordinate of the nth row.

[0123] The processing of steps S304 to S306 for the n+1th row is performed in the same or similar manner as the processing of the row whose v coordinate is the coordinate (v0). In the processing of step S304, the control unit 24 uses the parallax collection threshold calculated in the processing of step S306 for the nth row to collect the road surface candidate parallax d c In the process of step S305, the control unit 24 processes the collected road candidate parallax d c Averaging is performed to calculate the average road candidate parallax d av In the process of step S306, the control unit 24 uses the average road surface candidate parallax d av , changing the road surface height Y of the mathematical formula (1). The control unit 24 uses the mathematical formula (1) with the changed road surface height Y to calculate the geometrically estimated road surface parallax d s Furthermore, the control unit 24 estimates the road parallax d by geometrically estimating the road parallax d. s Considering the road height change ΔY, the disparity collection threshold is calculated to extract the road candidate disparity d in the n+2th row. c .

[0124] The control unit 24 sets the road candidate parallax d c The object to be extracted is the road candidate with the parallax d closest to the stereo camera 10. c The row corresponding to the extraction position of d is moved upward (negative direction of v coordinate) in sequence, while extracting the road candidate disparity d corresponding to the v coordinate. c The control unit 24 can extract the road candidate parallax d c and the average road surface candidate parallax d corresponding to the corresponding u coordinate, v coordinate, and v coordinate av Stored together in the memory 23.

[0125] In the process of step S307, the control unit 24 calculates the geometrically estimated road surface parallax d s When the road candidate parallax d is less than the above-mentioned predetermined value, the road candidate parallax d is terminated. c Extraction processing, return to Figure 7 The predetermined value may be, for example, 1 pixel.

[0126] In this way, Figure 8 In the flowchart, when observing from the stereo camera 10, the road candidate parallax d is extracted. cThe initial value of the v coordinate is set to v0 corresponding to the position on the near side, and the road surface candidate parallax d on the far side is extracted in sequence. c Generally, the stereo camera 10 has a higher detection accuracy of parallax at the close distance side than at the long distance side. Therefore, by sequentially extracting the road surface candidate parallax d from the close distance side to the long distance side, c , which can improve the detected road candidate parallax d c precision.

[0127] When extracting the above-mentioned road candidate parallax d c of Figure 8 In the flowchart, the road candidate parallax d is calculated for each longitudinal coordinate. c In other words, when extracting the above-mentioned road candidate parallax d c In the flowchart, the road candidate disparity d is calculated for each row of pixels in the vertical direction. c The unit for calculating the road candidate parallax is not limited to this. The road candidate parallax d may be calculated by aggregating a plurality of coordinates in the vertical direction. c .

[0128] Continue with steps S301 to S307 to calculate the road candidate parallax d c The extraction process of the control unit 24 enters Figure 7 The process of step S202 of the flowchart of FIG. r When the control unit 24 detects the road parallax d r The Kalman filter is applied sequentially. Therefore, the control unit 24 first initializes the Kalman filter (step S202). The Kalman filter calculated in step S305 can be used to perform the road parallax d r The average road surface candidate parallax d corresponding to the bottom row (row with v coordinate value v0) among the estimated rows av The value of is used as the initial value of the Kalman filter.

[0129] The control unit 24 sequentially executes the following processes of steps S203 to S210 while changing the target row from the short distance side to the long distance side of the road surface (step S203 ).

[0130] First, the control unit 24 calculates the target line in the first parallax image based on the parallax d of the road surface candidate within a range of a constant width in the real space. c , generating the road parallax d r The frequency histogram of each value of is obtained (step S204). The range of constant width in the actual space is a range that takes into account the width of the road lane. The constant width can be set to a value such as 2.5m or 3.5m. The range of parallax is obtained, for example, Figure 11The range is initially set to be surrounded by the solid frame line 45. The constant width is pre-stored in the memory 23 of the object detection device 20. By limiting the range of obtaining parallax to this range, the possibility of the control unit 24 mistaking objects other than the road surface 41A or buildings such as sound insulation walls for the road surface 41A can be reduced. As a result, the accuracy of road surface detection can be improved. As described later, the initial frame line 45 can be changed in sequence according to the conditions on the road ahead. Figure 11 The range of the obtained disparity is shown in the solid line.

[0131] For the target line, the control unit 24 calculates the road parallax d based on the Kalman filter. r The predicted value is used to set the road parallax d r The acquisition range of road parallax d r The acquisition range is based on the Kalman filter to predict the road parallax d of the next line r The reliability is determined by the variance σ of the Gaussian distribution. 2 (σ is the road parallax d r The control unit 24 can calculate the road parallax d by using the predicted value ±2σ or the like. r The control unit 24 obtains the range of the road candidate parallax d generated in the process of step S204. c The histogram of the road parallax d is set based on the Kalman filter. r The road parallax d with the highest extraction frequency within the acquisition range r The control unit 24 extracts the road parallax d r Let the road parallax d of the target row be r Observation value (step S205).

[0132] Next, the control unit 24 checks the road surface parallax d determined in the process of step S205. r is the correct road parallax d r , excluding parallax corresponding to objects (step S206). r The control unit 24 generates a map to the road parallax d r dv correlation diagram on the dv coordinate space with the v coordinate as the coordinate axis. When the road surface 41A is correctly detected, in the dv correlation diagram, as shown in FIG. Figure 12 As shown by the dashed line, the road parallax d r As the value of the v coordinate decreases, it also decreases linearly.

[0133] On the other hand, Figure 13As shown in FIG. 1 , when the parallax representing an object is mistakenly recognized as the parallax representing the road surface 41A, in the dv correlation diagram, the parallax d in the portion representing the object's parallax remains approximately constant regardless of changes in the longitudinal coordinate (v coordinate). Typically, since an object includes a portion perpendicular to the road surface 41A, the object is displayed in the first parallax image as including multiple parallaxes at equal distances. Figure 13 In the first portion R1, parallax d decreases as the v coordinate value changes. First portion R1 correctly detects the parallax representing road surface 41A. In the second portion R2, parallax d remains constant even when the v coordinate changes. Second portion R2 is considered to be the portion where parallax is mistakenly detected as representing an object. If a predetermined number of lines with approximately equal parallax d values persist, the control unit 24 can determine that parallax representing an object was mistakenly identified as parallax representing road surface 41A.

[0134] When it is determined in the process of step S206 that the parallax is not the correct road surface parallax d r When (step S206: No), the control unit 24 searches for the road surface parallax d again from the row determined to be the parallax indicating the object erroneously detected. r (Step S207). In the process of step S207, the control unit 24 searches the road parallax histogram again in the area of the row where the parallax d does not change even when the v coordinate value changes. If there is a high frequency of parallax in the area where the parallax is smaller than the parallax d determined in the process of step S205, the control unit 24 can determine that the parallax is the correct road parallax d. r Observed value of .

[0135] In the process of step S206, it is determined that the road parallax d r If it is correct (step S206: Yes), or the road parallax d is finished in the process of step S207, r When searching again, the control unit 24 proceeds to the process of step S208. In the process of step S208, the control unit 24 determines the horizontal range of the parallax image 41 corresponding to the road surface on the first parallax image to be the target of generating the next row of histograms shifted by 1 pixel in the vertical direction. For example, Figure 11 As shown, when the parallax image 42 corresponding to another vehicle exists on the parallax image 41 corresponding to the road surface, the road surface detection unit 26 cannot obtain the correct road surface parallax d of the parallax image 41 corresponding to the portion of the road surface that overlaps with the other vehicle. r When the road parallax d is obtained r When the range of the parallax image 41 corresponding to the road surface becomes narrow, it is difficult for the control unit 24 to obtain an accurate road surface parallax d r Therefore, if Figure 11As shown by the middle dashed line, the control unit 24 acquires the road candidate parallax d c Specifically, when it is determined in the process of step S206 that the parallax representing the object is included, the control unit 24 detects which side of the object represents the correct road parallax d. r The road candidate parallax d c In the next row, the control unit 24 makes the range of obtaining parallax gradually move to the horizontal direction to include more parallax d indicating the correct road surface. r The road candidate parallax d c One side ( Figure 11 Move to the middle right).

[0136] Next, the control unit 24 uses the road surface parallax d of the current row determined in the process of step S205 or the process of S207. r , update the Kalman filter (step S209). That is, the Kalman filter is based on the road parallax d of the current row. r Observation value, calculate the road parallax d r When calculating the estimated value of the current row, the control unit 24 adds the road parallax d of the current row. r The estimated value is taken as part of the past data and used to calculate the road parallax d of the next line r The estimated value is processed. It is assumed that the height of the road surface 41A does not change rapidly up and down relative to the horizontal distance Z from the stereo camera 10. Therefore, in the estimation using the Kalman filter of this embodiment, it is estimated that the road surface parallax d r There is a road parallax d of the next row near r Thus, by limiting the control unit 24 to the road parallax d of the current line r The parallax range of the next row of histograms is generated near , thereby reducing the possibility of erroneous detection of objects other than the road surface 41A. In addition, the amount of calculation executed by the control unit 24 can be reduced, thereby speeding up the processing.

[0137] In the process of step S209, the road surface parallax d estimated by the Kalman filter is r If the parallax d is greater than the predetermined value, the control unit 24 returns to the process of step S203 and repeats the processes of steps S203 to S209. r If the value is less than or equal to a predetermined value (step S210), the control unit 24 proceeds to the next process (step S211). The predetermined value may be, for example, 1 pixel.

[0138] In the process of step S211, the control unit 24 approximates the longitudinal image coordinate v and the estimated road surface parallax d using two straight lines on the dv correlation diagram. r The relationship between the road parallax d r The value of the v coordinate is related to the distance Z from the stereo camera 10 and the road surface height Y. Therefore, two straight lines are used to approximate the v coordinate and the road surface parallax d r The relationship between means that the relationship between the distance Z from the stereo camera 10 and the height of the road surface 41A can be approximated by two straight lines. Figure 14 The processing of step S211 is described in detail in the flowchart of FIG.

[0139] First, through the Figure 7 The process up to step S210 obtains the road parallax d r The correlation between v coordinate and road parallax d r The correlation between them is as follows in the dv coordinate space: Figure 15 Graph 51 is represented as a dashed line in the dv coordinate space. In real space, if road surface 41A is flat and has no inclination changes, graph 51 is a straight line. However, in real-world road surface 41A, the inclination of road surface 41A may change due to ups and downs. If the inclination of road surface 41A changes, graph 51 in the dv coordinate space cannot be represented by a straight line. If the inclination changes of road surface 41A are approximated using three or more straight lines or curves, the processing load on object detection device 20 increases. Therefore, in this application, curve 51 is approximated using two straight lines.

[0140] like Figure 15 As shown, the control unit 24 uses the first straight line 52 to calculate the estimated road surface parallax d on the lower side (short distance side) in the dv coordinate space based on the least square method. r Approximation is performed (step S401). Approximation using the first straight line 52 can be performed up to a road surface parallax d corresponding to a predetermined distance within the distance range targeted for object detection by the object detection device 20. r The predetermined distance can be set to half the distance range of the object detection target of the object detection device 20. For example, if the object detection device 20 is designed to detect objects up to 100 meters away, the first straight line 52 can be determined by the least squares method to be closest to the figure 51 within the range from the closest distance measurable by the stereo camera 10 to 50 meters away.

[0141] Next, the control unit 24 determines whether the inclination of the road surface 41A represented by the first straight line 52 approximated in step S401 is a possible inclination of the road surface 41A (step S402). The inclination angle of the first straight line 52 becomes a plane when transformed into real space. The inclination of the first straight line 52 corresponds to the inclination angle of the road surface 41A in the yz plane, determined based on conditions such as the road surface height Y0 at the installation position of the stereo camera 10 and the baseline length B. When the inclination of the road surface 41A in real space corresponding to the first straight line 52 is within a predetermined angle range relative to the horizontal plane in real space, the control unit 24 can determine that the inclination of the road surface 41A in real space corresponding to the first straight line 52 is a possible inclination. When the inclination of the road surface 41A in real space corresponding to the first straight line 52 is outside the predetermined angle range relative to the horizontal plane in real space, the control unit 24 can determine that the inclination of the road surface 41A in real space corresponding to the first straight line 52 is an impossible inclination. The predetermined angle can be appropriately set based on the driving environment of the mobile object 30.

[0142] In the process of step S402, if it is determined that the inclination of the first straight line 52 is an inclination that cannot exist as the road surface 41A (step S402: No), the control unit 24 determines the first straight line 52 based on a theoretical road surface assuming that the road surface 41A is flat (step S403). The theoretical road surface can be calculated based on the installation conditions such as the road surface height Y0, the installation angle, and the baseline length B at the installation position of the stereo camera 10. The road surface parallax d calculated from the image is r If the road parallax is not reliable, the control unit 24 uses the theoretical road parallax. For example, if the parallax representing an object or building other than the road surface 41A is mistakenly extracted as the road parallax d r In the case of , the control unit 24 determines that the road surface 41A has an unrealistic inclination, and can eliminate the error. This can reduce the possibility of misjudging the parallax representing objects or buildings other than the road surface 41A as the road surface parallax d. r possibility.

[0143] If, in step S402, the control unit 24 determines that the inclination of the first straight line 52 is a possible inclination of the road surface 41A (step S402: Yes), or after executing step S403, the control unit 24 proceeds to step S404. In step S404, the control unit 24 determines the approximate starting point 53 from which to begin approximation of the second straight line 55. The control unit 24 can calculate the approximation error with respect to the graph 51 in order from the smallest v-coordinate of the first straight line 52 (the far side) to the largest v-coordinate (the near side), and define the coordinates on the first straight line 52 where the approximation error is continuously less than a predetermined value as the approximate starting point 53. Alternatively, the control unit 24 can calculate the approximation error with respect to the graph 51 in order from the largest v-coordinate of the first straight line 52 (the near side) to the smallest v-coordinate (the far side), and define the approximate starting point 53 as the coordinate on the first straight line 52 where the approximation error is greater than the predetermined value. The v-coordinate of the approximate starting point 53 is not fixed to a specific value. Approximate starting point 53 can be set on first straight line 52 at a position corresponding to the v-coordinate closer to the position on the stereo camera 10 side than half the distance range for object detection by object detection device 20. For example, if first straight line 52 approximates road surface 41A within a range from the closest measurable distance to 50 meters, approximate starting point 53 can be set at a position corresponding to the v-coordinate 40 meters ahead of 50 meters.

[0144] After executing the process of step S404, the control unit 24 repeatedly executes the processes of steps S405 to S407. Figure 16 As shown, the control unit 24 uses the angle difference from the first straight line 52 as an angle to be selected from a predetermined angle range and sequentially selects candidate straight lines 54, which are candidates for the second straight line 55 starting from the approximate opening point 53 (step S405). The predetermined angle range is set to the angle within which the road slope can vary within the distance range to be measured. The predetermined angle range can be, for example, ±3 degrees. For example, the control unit 24 can change the angle of the candidate straight line 54 starting from -3 degrees relative to the first straight line 52, and then in increments of 0.001 degrees until the angle reaches +3 degrees relative to the first straight line 52.

[0145] For each selected candidate straight line 54, the control unit 24 calculates the error of the portion above (farther away from) the approximate starting point 53 of the graph 51 in the dv coordinate space (step S406). The error can be calculated using the mean squared error of the parallax d relative to the v coordinate. The control unit 24 may store the calculated error for each candidate straight line 54 in the memory 23.

[0146] When the calculation of the errors for all candidate straight lines 54 within the angle range is completed (step S407), the control unit 24 searches for the minimum error from the errors stored in the memory 23. Figure 17 As shown, the control unit 24 selects the candidate straight line 54 having the smallest error as the second straight line 55 (step S408).

[0147] When the second straight line 55 is determined in the process of step S408, the control unit 24 determines whether the error between the graph 51 and the second straight line 55 is within a predetermined value (step S409). The predetermined value is appropriately set to obtain the desired road surface estimation accuracy.

[0148] In the process of step S409, when the error is within the prescribed value (step S409: Yes), the road surface parallax d is approximated using the first straight line 52 and the second straight line 55. r .

[0149] In the process of step S409, if the error exceeds the predetermined value (step S409: No), the control unit 24 extends the first straight line 52 upward (far distance side) and rewrites the approximation result (step S410). As described above, the road surface parallax d is approximated using two straight lines. r .

[0150] Use two straight lines to approximate the road parallax d relative to the v coordinate r , thereby approximating the shape of the road surface using two straight lines. This reduces the subsequent computational load and speeds up object detection compared to using a curve or three or more straight lines to approximate the road surface shape. Furthermore, compared to using a single straight line to approximate the road surface, the error with the actual road surface is smaller. Furthermore, since the v coordinate of the approximate starting point 53 of the second straight line 55 is not fixed to a predetermined coordinate, the accuracy of the approximation with the actual road surface can be improved compared to a case where the coordinates of the approximate starting point 53 are fixed in advance.

[0151] In the process of step S409, when the error is within the prescribed value (step S409: Yes) or after executing the process of step S410, the control unit 24 ends the process of approximating the road surface parallax d using a straight line. r Processing, return to Figure 7 The processing of step S212.

[0152] In the process of step S212, the road parallax d removed from the first parallax image is determined. r The road parallax d removed from the first parallax image is r The threshold value is equivalent to the first height described later. The road parallax d can be removed in the next step S103. r way to calculate the first height.

[0153] Next, the control unit 24 returns to Figure 5 Through the above processing, the control unit 24 obtains the v coordinate and the road parallax d in the dv coordinate space approximated by two straight lines. r The approximate formula of the relationship between v coordinate in dv coordinate space and road parallax d r The relationship between the distance Z in front of the stereo camera 10 and the road height Y in the actual space can be obtained by using an approximate formula. The control unit 24 performs the second processing (step S103) based on the approximate formula. The second processing is to remove the parallax corresponding to the range below the first height from the road surface 41A in the actual space and the parallax corresponding to the subject above the second height from the road surface 41A from the first parallax image. Thus, the control unit 24 performs the second processing based on the approximate formula. Figure 6 The first parallax image 40 shown is generated Figure 18 The second parallax image 60 is shown. Figure 18 The figure is drawn for explanation. The actual second parallax image based on the image obtained from the stereo camera 10 is as follows. Figure 19 As shown. Figure 19 In the image, the size of the parallax is expressed by the shades of black and white. Figure 19 The second parallax image shown includes parallax images 44 corresponding to other vehicles.

[0154] The first height may be set to be smaller than the minimum value of the height of the object to be detected by the object detection device 20. The minimum value of the height of the object to be detected by the object detection device 20 may be the height of a child (e.g., 50 cm). The first height may be a value greater than 15 cm and less than 50 cm. Figure 6 The first parallax image 40 shown contains noise, and the aforementioned processing may reduce the accuracy of detecting the parallax corresponding to the road surface. In this case, if only the parallax corresponding to the detected road surface is removed from the first parallax image 40, the parallax corresponding to the road surface may remain in a portion of the parallax image 41. By removing the parallax corresponding to the range below the first height from the road surface 41A from the first parallax image 40, a second parallax image can be obtained in which the parallax corresponding to the road surface is removed from the parallax image 41 with high accuracy.

[0155] exist Figure 18In the second parallax image 60 shown, parallax information corresponding to the range below the first height from the road surface in real space is removed. With this structure, parallax information is not included in the parallax image 41 corresponding to the road surface. The parallax image 41 corresponding to the road surface and the parallax image 42 corresponding to other vehicles are adjacent. Since the second parallax image 60 does not include parallax information in the parallax image 41, the processing of the parallax information of the parallax image 42 corresponding to other vehicles can be facilitated in subsequent processing. Furthermore, by removing unnecessary parallax that is not related to the object to be detected, the processing speed described later can be increased.

[0156] The second height can be appropriately set based on the maximum value of the height of the object to be detected by the object detection device 20. When the mobile body 30 is a vehicle, the second height can be set based on the upper limit of the height of the vehicle that can be driven on the road. The height of the vehicle that can be driven on the road is stipulated by traffic laws and regulations. For example, in Japan's Road Traffic Law, the height of a truck is generally 3.8m or less. In this case, the second height can be 4m. By removing the parallax corresponding to the object whose height from the road surface is higher than the second height from the first parallax image, the information of the parallax corresponding to the object is removed from the first parallax image. Figure 18 In the second parallax image 60 shown, the parallax corresponding to the object at a height of the second height or higher from the road surface is removed, so that the parallax image corresponding to the object located on the negative side of the v axis does not contain parallax information. By removing the parallax information corresponding to the object at a height of the second height or higher, in the subsequent processing, Figure 18 The processing of the parallax information of the parallax image 42 corresponding to other vehicles can be performed more easily. Furthermore, by removing unnecessary parallax that is not related to the object to be detected, the processing speed described later can be increased.

[0157] After executing the process of step S103, the control unit 24 detects the first parallax and the second parallax from the second parallax image (step S104). Figure 20 The details of the process of step S104 are shown in the flowchart shown.

[0158] In the process of step S501, the control unit 24 divides the second parallax image into a plurality of partial areas by dividing it by Δu1 along the u direction. Figure 21, the partial area 61 is represented overlapping with the second parallax image 60. The partial area 61 can be a rectangle whose long side is significantly longer than the short side. The long side of the partial area 61 is more than 5 times the short side of the partial area 61. The long side of the partial area 61 can be more than 10 times the short side of the partial area 61, or more than 20 times the short side of the partial area 61. The short side of the partial area 61, i.e., Δu1, can be from a few pixels to dozens of pixels. As described later, the first parallax and the second parallax can be detected based on the partial area 61. The shorter the short side of the partial area 61, i.e., Δu1, the higher the resolution of the detection of the first parallax and the second parallax. The control unit 24 can be from Figure 21 The partial regions 61 are sequentially acquired from the negative direction side toward the positive direction side of the u-axis shown, and the following processes of step S502 to step S506 are executed.

[0159] In the process of step S502, the control unit 24 generates a parallax histogram for each partial area. Figure 22 An example of a disparity histogram is shown in . Figure 22 The horizontal axis corresponds to the magnitude of the parallax. Figure 22 The vertical axis corresponds to the frequency of parallax. The frequency of parallax is the number of parallax pixels representing the parallax contained in the partial area. Figure 21 As shown, in the partial area 61, multiple parallax pixels in the parallax image corresponding to the same object may be included. For example, the partial area 61-1 includes multiple parallax pixels 42a. The multiple parallax pixels 42a are the parallax pixels included in the partial area 61-1 among the multiple parallax pixels in the parallax image 42 corresponding to other vehicles. In the partial area 61, the parallax represented by the multiple parallax pixels corresponding to the same object may be of the same degree. For example, in the partial area 61-1, the parallax represented by the multiple parallax pixels 42a may be of the same degree. That is, in Figure 22 In the disparity histogram shown, the frequency of disparities corresponding to the same object can be increased.

[0160] In the process of step S502, the control unit 24 may Figure 22The width of the interval Sn of the disparity histogram shown expands as the parallax becomes smaller. Interval Sn is the interval of the nth disparity histogram counted from the side with smaller parallax. The starting point of interval Sn is disparity dn-1. The end point of interval Sn is parallax dn. For example, the control unit 24 can make the width of interval Sn-1 wider than the width of interval Sn by about 10% of the width of interval Sn. In the case where the distance from the stereo camera 10 to the object is long, the parallax corresponding to the object may become smaller than the case where the distance from the stereo camera 10 to the object is short. In the case where the distance from the stereo camera 10 to the object is long, the number of pixels occupied by the object on the stereo image may become smaller than the case where the distance from the stereo camera 10 to the object is short. That is, in the case where the distance from the stereo camera 10 to the object is long, the number of parallax pixels representing the parallax corresponding to the object on the second parallax image can be reduced compared to the case where the distance from the stereo camera 10 to the object is short. As the parallax becomes smaller, Figure 22 The width of the disparity histogram bin Sn is widened, making it easier to detect the disparity corresponding to an object at a long distance from the stereo camera 10 through the processing of steps S503 to S506 described later. The control unit 24 may sequentially execute the processing of steps S503 to S506 for each bin of the disparity histogram.

[0161] In step S503, the control unit 24 determines whether there is a segment Sn in the generated disparity histogram where the frequency of disparity exceeds the first threshold Tr1 (a predetermined threshold). If the control unit 24 determines that there is a segment Sn where the frequency of disparity exceeds the first threshold Tr1 (step S503: Yes), the control unit 24 detects the disparity from the start point of segment Sn (disparity dn-1) to the end point of segment Sn (disparity dn) as the first disparity (step S504). The control unit 24 detects the first disparity as if it corresponds to the object. The control unit 24 associates the detected first disparity with the u-coordinate and stores it in the memory 23. On the other hand, if the control unit 24 determines that there is no segment Sn in the generated disparity histogram where the frequency of disparity exceeds the first threshold Tr1 (step S503: No), the process proceeds to step S505.

[0162] The first threshold Tr1 can be set based on the minimum value of the height of the object to be detected by the object detection device 20 (for example, the height of a child is 50 cm). However, as described above, when the object is at a distance far from the stereo camera 10, the parallax corresponding to the object can be smaller than when the object is at a distance close to the stereo camera 10. In this case, if the first threshold Tr1 is kept constant with respect to the parallax, it may be difficult to detect the parallax corresponding to the object at a distance far from the stereo camera 10 compared to the object at a distance close to the stereo camera 10. Therefore, based on the minimum value of the height of the object to be detected by the object detection device 20, the first threshold Tr1 can be set to become larger as the parallax of the disparity histogram becomes smaller. For example, the first threshold Tr1 can be calculated by mathematical formula (2).

[0163] Tr1=(D×H) / B (2)

[0164] In equation (2), disparity D is the disparity corresponding to the horizontal axis of the disparity histogram. Height H is the minimum height of the object to be detected by object detection device 20. Baseline length B is the distance (baseline length) between the optical centers of first camera 11 and second camera 12.

[0165] The process of step S505 is to determine whether to detect the second parallax. Before explaining the process of step S505, the reason for detecting the second parallax is explained. As described above, in the matching process of calculating the parallax, due to the small amount of features on the stereo image, the parallax of a part of the parallax image corresponding to the object may not be calculated or may be smaller than the parallax of other parts. For example, Figure 19 As shown, due to the small number of feature quantities on the stereoscopic image, the parallax of the lower central part of the parallax image 44 corresponding to other vehicles cannot be calculated. If a part of the parallax of the parallax image corresponding to the object is not calculated, then, for example, even if the actual height of the object in the actual space is more than 50 cm, it can be displayed as two or more independent objects with a height of less than 50 cm (for example, 10 cm) in the actual space on the second parallax image. That is, even if there is an object whose actual height in the actual space is higher than the height of the detection target, if a part of the parallax image corresponding to the object is missing, the parallax corresponding to the object may not be detected as the first parallax through the judgment process of the above-mentioned first threshold Tr1. Therefore, in the present embodiment, this parallax that is not detected as the first parallax is detected as a candidate for the parallax corresponding to the object, that is, the second parallax. By detecting it as the second parallax, it is possible to Figure 5 In the process of step S107 described later, it is determined whether the parallax of the object to be detected is restored.

[0166] In the process of step S505 , the control unit 24 determines whether there is a section Sn in the generated disparity histogram where the frequency of disparity is equal to or less than the first threshold Tr1 and exceeds the second threshold Tr2 (predetermined threshold).

[0167] The second threshold Tr2 may be a predetermined ratio of the first threshold Tr1. The predetermined ratio is appropriately set based on the ratio between the portion where the parallax is not calculated and the portion where the parallax is calculated in the parallax image for the same object. For example, the predetermined ratio may be 0.2.

[0168] In the process of step S505, if the control unit 24 determines that there is a section Sn in which the frequency of parallax is less than or equal to the first threshold value Tr1 and exceeds the second threshold value Tr2 in the generated parallax histogram (step S505: Yes), the process proceeds to step S506. On the other hand, if the control unit 24 determines that there is no section Sn in which the frequency of parallax is less than or equal to the first threshold value Tr1 and exceeds the second threshold value Tr2 in the generated parallax histogram (step S505: No), the process returns to step S506. Figure 5 The process of step S105 is shown.

[0169] In the process of step S506, the control unit 24 detects the parallax in the range from the starting point of the interval Sn (parallax dn-1) to the end point of the interval Sn (parallax dn) as the second parallax (step S506). The control unit 24 associates the detected second parallax with the u coordinate and stores it in the memory 23. After executing the process of step S506, the control unit 24 returns to the Figure 5 The process of step S105 is shown.

[0170] In the process of step S105, the control unit 24 calculates the height of the object on the image. Figures 23 to 25 The details of the process of step S105 are shown in the flowchart shown.

[0171] Next, the control unit 24 performs a calculation process for calculating the height of the object on the second disparity map, i.e., the second disparity image, which is a disparity map. However, the control unit 24 may perform a calculation process for calculating the height of the object on any disparity map in which the disparity is associated with the uv coordinate. In the calculation process for the height of the object, the control unit 24 calculates the height of the object on the image by scanning the disparity pixels along the v direction in the u coordinate of the search object. For example, the control unit 24 calculates the height of the object on the image by scanning the disparity pixels representing the first disparity considered to correspond to the object and / or the disparity image representing the second disparity that is a candidate for the disparity corresponding to the object along the v direction.

[0172] When the process of step S601 begins, the control unit 24 obtains the minimum u coordinates as the u coordinates of the search target. In this embodiment, the minimum and maximum coordinates, when used for scanning on an image, refer to the minimum and maximum coordinates of the scanning range. The minimum and maximum coordinates of the scanning range do not necessarily coincide with the minimum and maximum coordinates on the image. The minimum and maximum coordinates of the scanning range can be set arbitrarily.

[0173] In the process of step S601, the control unit 24 refers to the memory 23 to determine whether there is a corresponding u coordinate of the search object. Figure 20 The processing shown detects the first disparity or the second disparity.

[0174] exist Figure 26 The second parallax image 70 is shown in FIG. Figure 26 A portion of the second parallax image 70 is shown in FIG. The second parallax image 70 corresponds to Figure 18 Region 62 of second parallax image 60 is shown. Second parallax image 70 includes parallax image 42 of the other vehicle. In second parallax image 70, the shaded portion represents parallax pixels representing the first parallax or the second parallax. When the u-coordinate of the search object is coordinate (u0), control unit 24 determines that there is no first parallax or second parallax associated with the u-coordinate of the search object. When the u-coordinate of the search object is coordinate (u1), control unit 24 determines that there is a first parallax or second parallax associated with the u-coordinate of the search object.

[0175] In step S601, if the control unit 24 does not determine that the first parallax or second parallax associated with the u-coordinate of the search object exists (step S601: No), the process proceeds to step S602. On the other hand, if the control unit 24 determines that the first parallax or second parallax associated with the u-coordinate of the search object exists (step S601: Yes), the process proceeds to step S604.

[0176] In the process of step S602, the control unit 24 determines whether the u coordinate of the search object is the maximum coordinate. When the u coordinate of the search object is the maximum coordinate, the control unit 24 has completely searched the u coordinate from the minimum coordinate to the maximum coordinate. When the control unit 24 determines that the u coordinate of the search object is the maximum coordinate (step S602: Yes), the process ends. Figure 23 The process shown returns to Figure 5 For example, when the u coordinate of the search object is Figure 26When the coordinate (u0) shown in FIG6 is obtained, the control unit 24 determines that the u coordinate of the search object is not the maximum coordinate. On the other hand, when the control unit 24 determines that the u coordinate of the search object is not the maximum coordinate (step S602: No), the u coordinate of the search object is increased by 1 (step S603). The control unit 24 performs the process of step S601 on the u coordinate obtained by increasing the u coordinate by 1 in the process of step S603. For example, the control unit 24 can Figure 26 The processes of steps S601 to S603 are repeatedly executed for the u coordinate of the search object starting from the coordinate (u0) shown in FIG. 1 until it is determined that there is a first parallax or a second parallax associated with the u coordinate of the search object.

[0177] In the process of step S604, the control unit 24 obtains the object parallax. The object parallax may be only the first parallax detected as the parallax corresponding to the object. In other words, the object parallax may be the parallax that satisfies the prescribed conditions. Figure 20 Alternatively, the object parallax may include both the first parallax and the second parallax. In other words, the object parallax may be the parallax that satisfies the prescribed condition. Figure 20 The parallax of the determination process of step S503 and the determination process of step S505 shown in FIG. Figures 23 to 25 In the illustrated processing, a case where the target parallax includes both the first parallax and the second parallax will be described.

[0178] In the process of step S604 , the control unit 24 acquires the largest parallax as the target parallax among the first parallax and the second parallax associated with the u-coordinate of the search target stored in the memory 23 .

[0179] In the process of step S605, the control unit 24 calculates the v coordinate of the road surface corresponding to the object parallax. For example, the control unit 24 substitutes the object parallax into the coordinates of the road surface represented by Figure 7 The v coordinate of the road surface and the road surface parallax d obtained in the process of step S211 are shown as follows: r The approximate relationship between Figure 17 ) of the road parallax d r To calculate the v coordinate of the road surface. Figure 26 In the example, it is assumed that the u coordinate of the search target is the coordinate (u1). In this case, the control unit 24 calculates the coordinate (v1) as the v coordinate of the road surface.

[0180] In step S606, the control unit 24 determines whether, based on the calculated v-coordinate of the road surface, there is a coordinate corresponding to a parallax approximately equal to the object parallax within a predetermined range along the negative v-axis. In the present invention, "parallax approximately equal to the object parallax" includes both parallaxes that are identical to the object parallax and parallaxes that are substantially identical to the object parallax. In the present invention, "parallax substantially equal to the object parallax" refers to parallaxes that can be treated as object parallaxes in image processing. For example, parallaxes that differ from the object parallax by within a range of ±10% can be considered to be approximately equal to the object parallax. Furthermore, the predetermined range in step S606 can be appropriately set based on the height of the object rising from the road surface. Examples of objects rising from the road surface include street trees, pedestrian bridges, and traffic lights. These objects rising from the road surface are outside the detection targets of the object detection device 20. In other words, height calculation is not necessary for such objects rising from the road surface.

[0181] If, in step S606, the control unit 24 does not determine that a coordinate corresponding to a parallax substantially equal to the target parallax exists within a predetermined range along the negative v-axis from the road surface's v-coordinate (step S606: No), the process proceeds to step S607, where the object's height is not calculated. In step S607, the control unit 24 adds a scanned-complete flag to the target parallax acquired in step S604 and stored in memory 23. After executing step S607, the control unit 24 executes step S601 again. In the re-executed step S601, the control unit 24 determines whether a first parallax or second parallax is not marked as scanned-complete for the search object's u-coordinate. In the re-executed step S604, the control unit 24 acquires the largest parallax of the unmarked first and second parallaxes as the target parallax.

[0182] In the process of step S606, when the control unit 24 determines that there are coordinates corresponding to the parallax substantially equal to the object parallax within the prescribed range along the negative direction of the v-axis from the v-coordinate of the road surface (step S606: Yes), the process proceeds to step S608. In the process of step S608, the control unit 24 obtains the coordinates corresponding to the parallax substantially equal to the object parallax within the prescribed range along the negative direction of the v-coordinate of the road surface as the first coordinates. Figure 26In the example, it is assumed that the u coordinate of the search object is (u1). Furthermore, it is assumed that a parallax approximately equal to the object's parallax is associated with the coordinates (u1, v2) of the parallax pixel 71. In other words, it is assumed that the parallax pixel 71 represents a parallax approximately equal to the object's parallax. Furthermore, it is assumed that the v coordinate of the road surface (v0) and the v coordinate of the parallax pixel 71 (v1) are within a specified range. In this case, the control unit 24 acquires the coordinates (u1, v2) of the parallax pixel 71 as the first coordinate.

[0183] After executing the process of step S608, the control unit 24 determines whether a parallax substantially equal to the object parallax is associated with the coordinates obtained by reducing the v coordinate of the first coordinate by 1 (step S609). Figure 26 In the example, it is assumed that the first coordinate is the coordinate (u1, v2) of the parallax pixel 71. The coordinate obtained by reducing the v coordinate of the parallax pixel 71 by 1 is the coordinate (u1, v3) of the parallax pixel 72. It is assumed that the parallax pixel 72 represents a parallax that is approximately equal to the object parallax. In other words, it is assumed that the parallax that is approximately equal to the object parallax is associated with the coordinate (u1, v3) of the parallax pixel 72. In this case, the control unit 24 determines that the parallax that is approximately equal to the object parallax is associated with the coordinate (u1, v3) of the parallax pixel 72 obtained by reducing the v coordinate of the first coordinate (u1, v2) by 1.

[0184] In the process of step S609, if the control unit 24 determines that the parallax substantially equal to the object parallax is associated with the coordinate obtained by reducing the v coordinate of the first coordinate by 1 (step S609: Yes), the process proceeds to step S610. In the process of step S610, the control unit 24 updates the coordinate obtained by reducing the v coordinate of the first coordinate by 1 as the first coordinate. Figure 26 In the example, when the first coordinate is the coordinate (u1, v2) of the parallax pixel 71, the first coordinate is updated to the coordinate (u1, v3) of the parallax pixel 72 through the process of step S610. After executing the process of step S610, the control unit 24 returns to the process of step S609. For example, the control unit 24 repeatedly executes the process of step S609 and the process of step S610 until the first coordinate is updated to the coordinate (u1, v3) of the parallax pixel 72. Figure 26 The coordinates (u1, v4) of the parallax pixel 73 shown are updated to the first coordinates, and the coordinates (u1, v4) of the parallax pixel 73 are associated with a parallax that is substantially equal to the object parallax.

[0185] In the process of step S609, when the control unit 24 determines that the parallax substantially equal to the object parallax is not associated with the coordinate obtained by reducing the v coordinate of the first coordinate by 1 (step S609: No), it enters the Figure 24 The process of step S611 is shown. Figure 26, it is assumed that the first coordinate is the coordinate (u1, v4) of the parallax pixel 73. The coordinate obtained by reducing the v coordinate of the parallax pixel 73 by 1 is the coordinate (u1, v5) of the parallax pixel 74. The parallax pixel 74 represents a parallax smaller than the parallax that is approximately equal to the object parallax. In other words, the parallax that is approximately equal to the object parallax is not associated with the coordinate (u1, v5) of the parallax pixel 74. In this case, the control unit 24 determines that the parallax that is approximately equal to the object parallax is not associated with the coordinate (u1, v5) of the parallax pixel 74 obtained by reducing the v coordinate of the first coordinate (u1, v4) by 1.

[0186] In the process of step S611, the control unit 24 determines whether the v coordinate of the first coordinate is the minimum coordinate. When the v coordinate of the first coordinate is the minimum coordinate, it is when the control unit 24 completely scans the second parallax image along the negative direction of the v axis. When the control unit 24 determines that the v coordinate of the first coordinate is the minimum coordinate (step S611: yes), the control unit 24 enters the process of step S616. On the other hand, when the control unit 24 determines that the v coordinate of the first coordinate is not the minimum coordinate (step S611: no), the control unit 24 enters the process of step S612. For example, when the first coordinate is Figure 26 In the case of the coordinates (u1, v4) of the parallax pixel 73 shown, the control unit 24 determines that the v coordinate of the first coordinate (coordinate (v4)) is not the minimum coordinate.

[0187] Before explaining the processing of step S612, please refer to Figure 26 .exist Figure 26 , it is assumed that the first coordinate is the coordinate (u1, v4) of parallax pixel 73. Parallax pixel 75 is located on the negative side of the v-axis relative to parallax pixel 73. It is assumed that parallax pixel 75 represents a parallax that is approximately equal to the object's parallax. Parallax pixel 73 and parallax pixel 75 are part of parallax image 42 corresponding to another vehicle that is the same object. However, due to parallax variations, even parallax pixels included in a parallax image corresponding to the same object may be arranged separately in the v-direction, as in parallax pixel 73 and parallax pixel 75.

[0188] Therefore, in the process of step S612, the control unit 24 determines whether there is a coordinate corresponding to a parallax substantially equal to the object parallax within a prescribed interval from the v coordinate of the first coordinate along the negative direction of the v axis. The prescribed interval can be appropriately set based on the height of the object to be detected by the object detection device 20. For example, the prescribed interval can be appropriately set based on the height of the rear of the vehicle (for example, 80 cm). Figure 26, the interval in the v direction between the parallax pixel 73 and the parallax pixel 75 is within a predetermined interval. When the first coordinate is the coordinate (u1, v4) of the parallax pixel 73, the control unit 24 determines that there is a coordinate (u1, v6) of the parallax pixel 75 associated with a parallax substantially equal to the object parallax.

[0189] In the process of step S612, if the control unit 24 determines that there are coordinates corresponding to a parallax substantially equal to the object parallax within a predetermined interval along the negative direction of the v-axis from the first coordinate (step S612: Yes), the process proceeds to step S613. On the other hand, if the control unit 24 determines that there are no coordinates corresponding to a parallax substantially equal to the object parallax within a predetermined interval along the negative direction of the v-axis from the first coordinate (step S612: No), the process proceeds to step S615.

[0190] The purpose of the process of step S613 is to determine whether the object parallax and the parallax substantially equal to the object parallax in the process of step S612 are parallaxes corresponding to the same object. Figure 27 as well as Figure 28 , illustrating an example of a second parallax image.

[0191] exist Figure 27 The second parallax image 80 is shown in FIG. The second parallax image 80 is based on Figure 28 The first image 90 includes an image 91 corresponding to other vehicles, an image 92 corresponding to street trees, and an image 93 corresponding to buildings. Figure 27 As shown, the second parallax image 80 includes: Figure 28 The parallax image 81 corresponding to the image 91 shown, and Figure 28 The disparity image 82 corresponding to the image 92 shown and the Figure 28 The image 93 shown corresponds to the parallax image 85 .

[0192] like Figure 27 As shown, the parallax image 81 includes parallax pixels 83. The parallax image 82 includes parallax pixels 84. The u coordinates of the parallax pixels 83 and 84 are the same at the coordinate (u2). In addition, in the actual space, the distance from the stereo camera 10 to the Figure 28 The distance of the other vehicle corresponding to the image 91 shown is the same as the distance from the stereo camera 10 to the Figure 28 The distances of the street trees corresponding to the image 92 shown are approximately equal. Since these two distances are approximately equal, Figure 27 The disparity represented by the disparity pixel 83 of the disparity image 81 is Figure 27The parallax represented by the parallax pixels 84 of the parallax image 82 shown in FIG. 8 can be substantially equal. Figure 28 The image 92 shown corresponds to other vehicles and Figure 28 The distances between the street trees in the image 92 shown are relatively close. Figure 27 The interval between the parallax pixels 83 of the parallax image 81 and the parallax pixels 84 of the parallax image 82 in the v direction shown is within the predetermined interval described above in the process of step S612 .

[0193] exist Figure 27 In the illustrated configuration, when the u coordinate of the search object is set to coordinate (u2), the control unit 24 acquires the parallax pixel 83 and the parallax represented by the parallax pixel 84 as the object parallax. Furthermore, the control unit 24 may update the coordinate of the parallax pixel 83 to the first coordinate. When the u coordinate of the search object is coordinate (u2), it is preferable to calculate the height T1 corresponding to the parallax image 81 as the height of the object on the image. As an assumed example, when the first coordinate is the coordinate of the parallax pixel 83, it is assumed that the interval between the parallax pixel 83 and the parallax pixel 84 is within the above-mentioned prescribed interval, and thus the control unit 24 updates the coordinate of the parallax pixel 84 to the first coordinate. In this assumed example, the control unit 24 can continuously scan the parallax pixels of the parallax image 82 corresponding to the street trees. As a result, the control unit 24 detects the height relative to the parallax image 82, that is, the height T2 corresponding to the street trees, as the height of the object.

[0194] Among them, multiple parallax pixels 86 exist between parallax pixel 83 and parallax pixel 84. Multiple parallax pixels 86 are included in parallax image 85. The building corresponding to parallax image 85 is located farther away from stereo camera 10 than the vehicle and street trees. Therefore, the parallax represented by multiple parallax pixels 86 in parallax image 85 is smaller than the parallax represented by parallax pixels 83 and parallax pixels 84.

[0195] Therefore, in the process of step S613, the control unit 24 determines whether there are more than a predetermined number of coordinates between the first coordinate and the second coordinate that are associated with the third parallax. The second coordinate is a coordinate located within a predetermined interval along the negative direction of the v-axis from the first coordinate and is associated with a parallax that is substantially equal to the object parallax. For example, when the first coordinate is Figure 27 In the case of the coordinates of the parallax pixel 83 shown in FIG, the second coordinates become the coordinates of the parallax pixel 84. The third parallax is a parallax smaller than the object parallax. The third parallax can be set based on the parallax corresponding to the background. For example, it can be assumed that Figure 27The third parallax is set based on the parallax represented by the parallax pixels 86. The predetermined number of the predetermined intervals described above can be appropriately set assuming that the number of parallax pixels representing the third parallax is included.

[0196] In the process of step S613, when the control unit 24 determines that there are more than a predetermined number of coordinates corresponding to the third parallax between the first coordinate and the second coordinate (step S613: Yes), the process proceeds to step S616. Figure 27 In the case of the coordinates of the parallax pixel 83 and the parallax pixel 84 shown in FIG, the control unit 24 determines that there are more than a predetermined number of parallax pixels 86 between the coordinates of the parallax pixel 83 and the coordinates of the parallax pixel 84. On the other hand, when the control unit 24 determines that there are no more than a predetermined number of coordinates corresponding to the third parallax between the first coordinate and the second coordinate (step S613: No), the process proceeds to step S614. For example, when the first coordinate and the second coordinate are Figure 26 When the coordinates of the parallax pixel 73 and the coordinates of the parallax pixel 75 are shown, the control unit 24 determines that there are no parallax pixels 74 exceeding a predetermined number between the coordinates of the parallax pixel 73 and the coordinates of the parallax pixel 75 .

[0197] In the process of step S614, the control unit 24 updates the second coordinate to the first coordinate. For example, when the first coordinate and the second coordinate are Figure 26 When the coordinates of the parallax pixel 73 and the coordinates of the parallax pixel 75 are updated, the control unit 24 updates the coordinates (u1, v6) of the parallax pixel 75 to the first coordinates. After executing the process of step S614, the control unit 24 returns to Figure 23 For example, in step S609 Figure 26 In the illustrated configuration, the control unit 24 may repeatedly execute the process starting from step S609 until the coordinates (u1, v7) of the parallax pixel 76 are updated to the first coordinate position.

[0198] In the process of step S615, the control unit 24 determines whether there is a coordinate corresponding to a parallax substantially equal to the object parallax beyond a predetermined interval along the negative direction of the v axis from the first coordinate. For example, when the first coordinate is Figure 26When the coordinates (u1, v7) of the parallax pixel 76 shown are shown, the control unit 24 determines that there are no coordinates corresponding to the parallax that is approximately equal to the object parallax beyond the prescribed interval along the negative direction of the v-axis starting from the coordinates (u1, v7). When the control unit 24 determines that there are no coordinates corresponding to the parallax that is approximately equal to the object parallax beyond the prescribed interval along the negative direction of the v-axis starting from the first coordinate (step S615: No), the control unit 24 proceeds to the processing of step S616. On the other hand, when the control unit 24 determines that there are coordinates corresponding to the parallax that is approximately equal to the object parallax beyond the prescribed interval along the negative direction of the v-axis starting from the first coordinate (step S615: Yes), the control unit 24 proceeds to the processing of step S616. Figure 25 The processing of step S618 is shown.

[0199] In the process of step S616, the control unit 24 calculates the height of the object on the image by subtracting the v coordinate of the road surface from the v coordinate of the first coordinate. Figure 26 In the case of the coordinates (u1, v7) of the parallax pixel 76 shown in FIG, the control unit 24 calculates the height of the object by subtracting the v coordinate of the road surface (coordinate (v1)) from the v coordinate of the parallax pixel 76 (coordinate (v7)). Figure 27 When the coordinates of the parallax pixel 83 shown in FIG. 8 are obtained, the control unit 24 calculates the height T1 of the object by subtracting the v coordinate of the road surface (coordinate (v8)) from the v coordinate of the parallax pixel 83. Figure 27 As shown, through the processing of steps S613 and S616, even if the parallax image 81 of the other vehicle and the parallax image 82 of the street tree are located relatively close, the height of the other vehicle can be calculated with high accuracy. In the processing of step S616, the control unit 24 associates the calculated height of the object in the image with the first coordinate and stores it in the memory 23.

[0200] In step S616, the control unit 24 may convert the calculated height of the object in the image into the height of the object in real space. If the height of the object in real space is less than the minimum height of the object to be detected, the control unit 24 may discard the calculated height information. For example, if the minimum height of the object to be detected is 50 cm, the control unit 24 may discard the calculated height information if the height of the object in real space is less than 40 cm.

[0201] After executing the process of step S616, the control unit 24 executes the process of step S617. In the process of step S617, the control unit 24 adds a scanned mark to the object parallax. After executing the process of step S617, the control unit 24 returns to Figure 23 The process of step S601 is shown.

[0202] Before explaining the process of step S618, refer to Figure 29 An example of the second parallax image is described below. Figure 29 The second parallax image 100 is shown in FIG. Figure 29 A portion of the second parallax image 100 is shown in FIG. The parallax image 101 is based on Figure 30 The parallax image generated by the first image 106 is shown. The first image 106 includes an image 107 corresponding to the back of the truck.

[0203] like Figure 29 As shown, the second parallax image 100 includes a parallax image 101. The parallax image 101 and Figure 30 The parallax image 101 includes parallax pixels 102, 103, 104, and 105. The parallax pixels 103 and 105 correspond to each other. Figure 30 The lower image 107a of the image 107 shown in FIG. The parallax pixels 104 and 105 correspond to Figure 30 The parallax pixels 103, 104, and 105 correspond to the upper image 107b of the illustrated image 107. The parallax pixels 103, 104, and 105 correspond to the same object, the truck, and are therefore substantially equal.

[0204] like Figure 29 As shown, multiple disparity pixels 102 are located in the center of disparity image 101. Disparity is not calculated for these disparity pixels 102. In other words, these disparity pixels 102 do not include disparity information. Generally speaking, the number of features in the stereo image is relatively small in the center of the rear of the truck. Because the number of features in the stereo image is relatively small in the center of the rear of the truck, the above-described matching process for calculating disparity may not calculate disparity, as is the case with disparity pixels 102.

[0205] exist Figure 29 In the illustrated structure, as an example, consider an example where the control unit 24 scans along the v direction using the u coordinate of the search object as the coordinate (u3). In this example, it is preferable to calculate the height T3 corresponding to the parallax image 101, that is, the height of the truck, as the height of the object. In this example, the control unit 24 can obtain the parallax represented by the parallax pixel 103 as the object parallax. In addition, the control unit 24 uses the coordinate (u3, v 10 ) is updated to the first coordinate. In order to calculate the height T3, in the above-mentioned step S614, the coordinates (u3, v 11 ) is updated to the first coordinate. In order to update the coordinates (u3, v 11) is updated to the first coordinate, it is necessary to make the predetermined interval in the above-mentioned step S612 larger than the v coordinate (coordinate (v 11 )) and the v coordinate of the parallax pixel 103 (coordinate (v 10 However, if the predetermined interval in the process of the above-mentioned step S612 is enlarged, the possibility of miscalculating the height of the object increases.

[0206] Therefore, the control unit 24 calculates the Figure 29 The height T4 shown is the first candidate height, and the calculation Figure 29 The height T3 shown is the second candidate height. Figure 5 In step S107, the control unit 24 determines which of the first candidate height and the second candidate height to use as the object's height. This configuration eliminates the need to increase the predetermined interval in step S612. This avoids increasing the predetermined interval in step S612, thereby reducing the likelihood of miscalculating the object's height.

[0207] In the process of step S618, the control unit 24 obtains candidate coordinates. Candidate coordinates are coordinates that exist beyond a predetermined interval along the negative direction of the v-axis from the first coordinate and are associated with a parallax that is approximately equal to the object parallax. For example, when the first coordinate is Figure 29 The coordinates (u3, v 10 ), the control unit 24 obtains the coordinates (u3, v 11 ) as candidate coordinates.

[0208] In the process of step S619, the control unit 24 calculates the first candidate height by subtracting the v coordinate of the road surface from the v coordinate of the first coordinate. Figure 29 The coordinates (u3, v 10 ) when the control unit 24 obtains the v coordinate (coordinate (v 10 )) by subtracting the v coordinate of the road surface (coordinate (v9)) to calculate the first candidate height T4. In the process of step S619, the control unit 24 associates the calculated first candidate height with the first coordinate and stores it in the memory 23. For example, the control unit 24 Figure 29 The first candidate height T4 shown is related to the coordinates (u3, v 10 ) are stored in memory 23 in association.

[0209] In the process of step S620, the control unit 24 determines whether the parallax that is approximately equal to the object parallax is associated with the coordinates obtained by reducing the v coordinate of the candidate coordinates by 1. When the control unit 24 determines that the parallax that is approximately equal to the object parallax is associated with the coordinates obtained by reducing the v coordinate of the candidate coordinates by 1 (step S620: yes), the control unit 24 proceeds to the process of step S621. In the process of step S621, the control unit 24 updates the coordinates obtained by reducing the v coordinate of the candidate coordinates by 1 as the candidate coordinates. After executing the process of step S621, the control unit 24 returns to the process of step S620. For example, in Figure 29 In the structure shown, the control unit 24 repeatedly executes the process of step S620 and the process of step S621 until the coordinates (u3, v 12 ) is updated to the candidate coordinates. On the other hand, when the control unit 24 determines that the parallax substantially equal to the object parallax is not associated with the coordinates obtained by reducing the v coordinate of the candidate coordinates by 1 (step S620: No), the process proceeds to step S622. For example, when the candidate coordinates are Figure 29 The coordinates (u3, v 12 ), the control unit 24 determines that the parallax substantially equal to the object parallax is not associated with the coordinates obtained by reducing the v coordinate of the candidate coordinates by 1.

[0210] In the process of step S622, the control unit 24 calculates the second candidate height by subtracting the v coordinate of the road surface from the v coordinate of the candidate coordinate. Figure 29 The coordinates (u3, v 12 ) in the case of the control unit 24 by obtaining the v coordinate (coordinate (v 12 )) by subtracting the v coordinate of the road surface (coordinate (v9)) to calculate the second candidate height T3. In the process of step S622, the control unit 24 associates the calculated second candidate height with the candidate coordinate and stores it in the memory 23. For example, the control unit 24 Figure 29 The second candidate height T3 shown is related to the coordinates (u3, v 12 ) is associated and stored in the memory 23. After executing the process of step S622, the control unit 24 proceeds to the process of step S623.

[0211] In the process of step S623, the control unit 24 adds a scanned object parallax acquired in the process of step S604 stored in the memory 23. After executing the process of step S623, the control unit 24 returns to the Figure 23 The process of step S601 is shown.

[0212] In the process of step S106, the control unit 24 performs parallel object detection processing. Figure 31 The details of the process of step S106 are shown in the flowchart shown.

[0213] In the processing of step S701, the control unit 24 generates or acquires a UD map. The UD map is also called "U-disparity space" and "ud coordinate space". The UD map is a map that establishes a correspondence between two-dimensional coordinates consisting of the u direction and the d direction corresponding to the magnitude of the disparity and the object disparity. The object disparity may be only the first disparity detected as the disparity corresponding to the object. Alternatively, the object disparity may include both the first disparity and the second disparity. In the following, it is assumed that the object disparity includes the first disparity and the second disparity. The control unit 24 can acquire the first disparity and the second disparity stored in the memory 23 to generate the UD map. Alternatively, the control unit 24 can acquire the UD map from the outside through the acquisition unit 21.

[0214] exist Figure 32 The UD diagram 110 is shown in FIG. The horizontal axis of the UD diagram 110 corresponds to the u axis. The vertical axis of the UD diagram 110 corresponds to the d axis representing the magnitude of the parallax. The coordinate system composed of the u coordinate and the d coordinate is also called the "ud coordinate system". In the UD diagram 110, the direction Figure 32 The lower left corner of the paper is used as the origin (0, 0) of the ud coordinate system. Figure 32 The plotted points shown are those obtained by Figure 20 The first parallax and the second parallax detected by the process shown in the figure establish corresponding coordinate points. The coordinate points are also referred to as "points". The UD map 110 includes point group 111, point group 112, point group 113, point group 114, and point group 115. The UD map 110 is based on the Figure 33 The diagram is generated by generating a second parallax image based on the first image 120 shown.

[0215] like Figure 33 As shown, the first image 120 includes an image 121 corresponding to a guardrail, images 122 and 123 corresponding to other vehicles, an image 124 corresponding to a pedestrian, and an image 125 corresponding to a sidewall. The first disparity and the second disparity detected from each of the images 121 to 125 correspond to Figure 32 Each of the point groups 111 to 115 shown. Images 121 and 125 may be images corresponding to parallel objects. In other words, Figure 32 The point group 111 and the point group 115 shown may be the parallax corresponding to the parallel object. One of the purposes of the parallel object detection process is to detect the image Figure 32 Parallax corresponding to parallel objects such as the point group 111 and the point group 115 shown.

[0216] In this embodiment, the image is detected by applying Hough transform in the process of step S704 described later. Figure 32 The parallax corresponding to parallel objects such as the point group 111 and the point group 115 shown in FIG. Figure 32 As shown in FIG. 1 , point group 112 and point group 113 are located near point group 111 and point group 115. If point group 112 and point group 113 are located near point group 111 and point group 115, the accuracy of detecting point group 111 and point group 115 can be reduced by the Hough transform in the process of step S704 described later. Therefore, the control unit 24 determines whether there is such a situation. Figure 32 Point group 112 and point group 113 are located near point group 111 and point group 115 .

[0217] Specifically, in the process of step S702, the control unit 24 determines whether there is a point group substantially parallel to the u direction. Figure 32 As shown, point groups 112 and 113 located near point groups 111 and 115 are from Figure 33 The parallax of the images 122 and 123 corresponding to the other vehicles shown in FIG. The other vehicles include a portion that is substantially parallel to the width direction of the road surface. Figure 32 As shown in FIG, since the other vehicles include a portion that is substantially parallel to the width direction of the road surface, the point group 112 and the point group 113 may be substantially parallel to the u direction. Therefore, by determining whether there is a point group that is substantially parallel to the u direction, it is possible to determine whether there is a point group that is substantially parallel to the u direction. Figure 32 Point group 112 and point group 113 as shown.

[0218] As an example of the process of step S702, first, the control unit 24 scans along the u direction of the UD map. Figure 32 The UD diagram 110 shown is scanned from the negative direction of the u-axis to the positive direction of the u-axis. The control unit 24 scans along the u-direction of the UD diagram while determining whether there are points that are continuously arranged in the prescribed range along the u-direction. The prescribed range can be based on an image relative to the vehicle (for example, Figure 33 The control unit 24 determines that there are points that are continuously arranged in a predetermined range along the u direction as a point group that is substantially parallel to the u direction. The control unit 24 detects the parallax points that are continuously arranged in a predetermined interval along the u direction as a point group that is substantially parallel to the u direction. For example, the control unit 24 Figure 32 The point groups 112 and 113 shown are detected as point groups substantially parallel to the u direction.

[0219] In the process of step S702, if the control unit 24 determines that a point cluster substantially parallel to the u direction exists (step S702: Yes), the process proceeds to step S703. On the other hand, if the control unit 24 determines that a point cluster substantially parallel to the u direction does not exist (step S702: No), the process proceeds to step S704.

[0220] In the process of step S703, the control unit 24 removes the point group that is substantially parallel to the detected u direction from the UD map. Instead of removing the point group that is substantially parallel to the u direction from the UD map, the control unit 24 may exclude the point group that is substantially parallel to the u direction among the coordinate points of the UD map from the application of the Hough transform in the process of step S704 described later. Figure 34 UD diagram 110 is shown in FIG. 110 in which point groups approximately parallel to the u direction have been removed. Figure 34 In the UD diagram 110 shown, the Figure 32 Point group 112 and point group 113 are shown.

[0221] In the processing of step S704, the control unit 24 detects a straight line by applying the Hough transform to the points contained in the UD map. Among them, a straight line can also be detected by transforming the UD map into a coordinate system of the actual space composed of xz coordinates, and applying the Hough transform to the coordinate system of the actual space after the transformation. However, if the baseline length B between the first camera 11 and the second camera 12 is short, it may sometimes cause the intervals between the coordinate points contained in the coordinate system of the actual space to be sparser than the intervals between the coordinate points contained in the UD map. If the intervals between the coordinate points are sparse, sometimes a straight line cannot be detected with high precision by the Hough transform. In this embodiment, by applying the Hough transform to the UD map, a straight line can be detected with high precision even if the baseline length B between the first camera 11 and the second camera 12 is short.

[0222] Reference Figure 35 as well as Figure 36 An example of the processing of step S704 will be described.

[0223] exist Figure 35 A portion of the UD graph is shown in FIG. Points 131, 132, 133, and 134 are coordinate points on the UD graph. The Hough transform is explained using point 131 as an example. The uv coordinate of point 131 on the UD graph is the coordinate (u 131 , d 131 ). The straight line L1 passing through point 131 can be defined infinitely. For example, the length of the normal line from the origin (0, 0) of the uv coordinate to the straight line L1-1 is length r. This normal line is inclined at an angle θ from the u-axis toward the positive direction of the d-axis. The control unit 24 uses the length r and the angle θ as variables to obtain the following mathematical formula (3) as the general formula for the straight line L1 passing through point 131.

[0224] r=u 131 ×cosθ+d 131 ×sinθ (3)

[0225] The control unit 24 projects the equation of the straight line L1 (Equation (3)) onto Figure 36 As shown in the rθ plane of the Hough space. Figure 36 The horizontal axis of the rθ plane shown is the r-axis. Figure 36 The vertical axis of the rθ plane shown is the θ axis. Figure 36 The curve 131L shown is a curve expressed by the mathematical formula (3). The curve 131L is expressed as a sine curve in the rθ plane. Similar to the point 131, the control unit 24 obtains Figure 35 The control unit 24 projects the equation of the straight line passing through the points 132 to 134 as shown in FIG. Figure 36 As shown in the rθ plane of the Hough space. Figure 36 The curves 132L to 134L shown are respectively obtained by Figure 35 The straight line corresponding to the points 132 to 134 shown in FIG. Figure 36 As shown, curves 131L to 134L can be at point P L The control unit 24 obtains point P L The rθ coordinate (coordinate (θ L , r L )). The control unit 24 is based on point P L The coordinates (θ L , r L ), the test passed Figure 35 The control unit 24 detects the following mathematical formula (4) as the equation of the straight line between the points 131 to 134 shown in FIG. Figure 35 The straight line from point 131 to point 134 is shown.

[0226] rL=u×cosθ L +d×sinθ L (4)

[0227] By executing the process of step S704, Figure 34 As shown, the control unit 24 can detect a straight line 111L corresponding to the point group 111 and a straight line 115L corresponding to the point group 115 .

[0228] In addition, if Figure 33 As shown, the image 121 and the image 125 corresponding to the parallel object are directed toward the vanishing point 120. VP Extension. Figure 34 As shown, since the image 121 and the image 125 are moving toward the vanishing point 120 VPTherefore, the point group 111 and the point group 115 also extend toward the vanishing point 120. VP The corresponding vanishing point is 110 VP Since point group 111 and point group 115 also extend to vanishing point 110 VP Therefore, the straight line 111L and the straight line 115L can also be extended to the vanishing point 110 VP extend.

[0229] Therefore, in the process of step S704, when the control unit 24 obtains the equation of the straight line passing through the point on the UD graph, it can obtain the equation of the straight line passing through the specified range based on the vanishing point. Figure 35 When the equation of the straight line L1 passing through the point 131 is obtained, the straight line L1 passing through the point 131 and the predetermined range Δu can be obtained. VP The formula of the straight line. The specified range Δu VP It is the range based on the vanishing point. VP Point 135 is included. Point 135 may be a vanishing point when the moving object 30 is moving in a straight line. By making the parallax at infinity zero, the d coordinate of point 135 may be zero. Since the u coordinate of the vanishing point is half of the maximum u coordinate when the moving object 30 is moving in a straight line, the u coordinate of point 135 (coordinate u VP ) can be half of the maximum u coordinate. VP When the moving object 30 is traveling on a curve, Figure 34 The vanishing point 110 is shown VP The offset from point 135 is appropriately set. Figure 36 As shown, in the rθ plane of the Hough space, the range of the θ axis where the curves 131L to 134L are drawn can be narrowed to the range Δθ VP By narrowing the range of the θ axis where the curves 131L to 134L are plotted to the range Δθ VP , the amount of computation required for the Hough transform can be reduced. By reducing the amount of computation required for the Hough transform, the processing speed of the Hough transform can be increased.

[0230] In the process of step S705, the control unit 24 determines whether the length of the straight line detected by the process of step S704 exceeds a predetermined length. The predetermined length can be appropriately set based on the length of buildings, i.e., parallel objects, arranged along the road surface. If the control unit 24 determines that the length of the straight line is less than the predetermined length (step S705: No), it returns to the process of Figure 5On the other hand, when the control unit 24 determines that the length of the straight line exceeds the predetermined length (step S705: Yes), the control unit 24 adds a parallel object mark to the point group corresponding to the straight line (step S706). Figure 34 The point group 111 and the point group 115 shown in FIG. 11 are marked with parallel object labels. After executing the process of step S706, the control unit 24 enters the Figure 5 The process of step S107 is shown.

[0231] In the process of step S107, the control unit 24 performs a restoration process. Figure 37 The details of the process of step S107 are shown in the flowchart shown.

[0232] In the process of step S801, the control unit 24 performs Figure 31 The same or similar processing as step S701 is performed to generate or obtain a UD map. Figure 31 When a UD diagram is generated in the process of step S701 shown in FIG. Figure 31 The UD map generated in the process of step S701 is shown.

[0233] exist Figure 38 UD diagram 140 is shown in FIG. UD diagram 140 is an enlarged Figure 34 The horizontal axis of the UD graph 140 corresponds to the u axis. The vertical axis of the UD graph 140 corresponds to the d axis indicating the magnitude of the parallax. Figure 38 , the UD map 140 is shown as an image. The pixels of the UD map 140 represent parallax. The pixels marked with hatching are pixels representing the first parallax. The pixels representing the first parallax, that is, the coordinates corresponding to the first parallax are also called "first parallax points". The pixels marked with dots are pixels representing the second parallax. The pixels representing the second parallax, that is, the coordinates corresponding to the second parallax are also called "second parallax points". The UD map 140 includes: a first parallax point 141, a first parallax point 142, and a second parallax point 143. The UD map 140 is a map generated based on a second parallax image including a parallax image corresponding to the rear of the vehicle. Since the rear glass and the like are located in the central part of the rear of the vehicle, the feature amount on the stereoscopic image is small. In the central part of the rear of the vehicle, since the feature amount on the stereoscopic image is small, the parallax is smaller than that of other parts, such as Figure 19 Sometimes, the parallax information that can be obtained is reduced, as shown in the parallax image 44. Figure 38 As shown, there is a second parallax point 143 between the first parallax point 141 and the first parallax point 142 .

[0234] In the processing of step S802, in the u direction of the UD diagram, the control unit 24 determines whether the second parallax point sandwiched between the first parallax points exists beyond the prescribed range. For example, the control unit 24 scans from the negative direction side of the u axis of the UD diagram to the positive direction side of the u axis. The control unit 24 determines whether the second parallax point sandwiched between the first parallax points exists beyond the prescribed range in the u direction of the UD diagram by scanning along the u direction. The prescribed range can be appropriately set based on the width (for example, 1m) behind the vehicle in the actual space. In the case where the second parallax point sandwiched between the first parallax points exists beyond the prescribed range, these second parallax points are likely to be parallaxes corresponding to different objects, such as different vehicles running in parallel. In contrast, in the case where the second parallax point sandwiched between the first parallax points exists within the prescribed range, these second parallax points are likely to be parallaxes corresponding to the same object. In Figure 38 In the u direction of the UD image 140 shown, a second parallax point 143, which is sandwiched between the first parallax point 141 and the first parallax point 142, exists within the prescribed range. In the u direction of the UD image 140, the control unit 24 does not determine that the second parallax point 143, which is sandwiched between the first parallax point 141 and the first parallax point 142, exists beyond the prescribed range.

[0235] In the process of step S802, if the control unit 24 determines that the second parallax points sandwiched between the first parallax points exceed the predetermined range in the u direction of the UD image (step S802: Yes), the process proceeds to step S805. On the other hand, if the control unit 24 does not determine that the second parallax points sandwiched between the first parallax points exceed the predetermined range in the u direction of the UD image (step S802: No), the process proceeds to step S803.

[0236] In the process of step S803, the control unit 24 obtains the first parallax corresponding to the two first parallax points sandwiching the second parallax point. Figure 38 The first parallax point 141 shown in FIG. 14 is associated with the first parallax point 142. Furthermore, in the process of step S802, the control unit 24 determines whether the difference in height in the real space corresponding to each of the two first parallaxes is within a predetermined height. For example, the control unit 24 determines whether the difference in height in the real space corresponding to each of the two first parallaxes is within a predetermined height. Figure 38 Is the difference between the height in real space corresponding to the first parallax of the first parallax point 141 and the height in real space corresponding to the first parallax of the first parallax point 142 equal to or less than a predetermined height? The predetermined height can be appropriately set based on the height of the vehicle in real space (e.g., 1 m).

[0237] In step S803, if the control unit 24 determines that the difference in height in real space corresponding to each of the two acquired first parallaxes is within a predetermined range (step S803: Yes), the process proceeds to step S804. In step S804, the control unit 24 adds a restoration flag to any second parallax point located between the first parallax points and exceeding the predetermined range. As described above, the second parallax point is the ud coordinate representing the second parallax. In other words, step S804 can also be considered a process of adding a restoration flag to the second parallax and the u coordinate associated with it. Figure 39 The UD graph 140 to which the restoration mark is added is shown. Figure 39 The pixels marked with thick hatching are shown as second disparity points with restoration marks attached.

[0238] In the process of step S803 , when the control unit 24 determines that the difference in height in the real space corresponding to each of the two acquired first parallaxes exceeds a predetermined height (step S803 : No), the process proceeds to step S805 .

[0239] The processing after step S805 is to determine the Figure 25 The first candidate height calculated in the process of step S619 shown in FIG. Figure 25 Which of the second candidate heights calculated in the process of step S622 shown in FIG. Figure 40 and Figure 41 An example of the second parallax image will be described.

[0240] exist Figure 40 The second parallax image 150 is shown in FIG. Figure 41 The second parallax image 160 is shown in FIG. Figure 40 as well as Figure 41 FIG2 shows a portion of the second parallax image 150 and the second parallax image 160. In the second parallax image 150 and the second parallax image 160, white parallax pixels are pixels that do not contain parallax information. Parallax pixels marked with thicker hatching are pixels that contain parallax information. Parallax pixels marked with thinner hatching are pixels used to calculate Figure 25 The parallax pixel at the second candidate height in the process of step S622 shown in FIG. Figure 25 In the process of step S622 shown in FIG, the coordinates of the disparity pixels marked with thinner hatching are associated with the second candidate height as candidate coordinates. The disparity pixels marked with thick dots are used to calculate Figure 25 The parallax pixel at the first candidate height in the process of step S619 shown in FIG. Figure 25In the process of step S619 shown in FIG, the coordinates of the disparity pixels marked with thick circles are associated with the first candidate height as the first coordinates. The disparity pixels marked with thin circles are used to calculate Figure 24 The parallax pixels of the height of the object in the process of step S616 are shown. Figure 24 In the process of step S616 shown, the coordinates of the parallax pixels marked with thin circles are associated with the height of the object as first coordinates.

[0241] and Figure 29 The second parallax image 100 shown is similar, Figure 40 The second parallax image 150 is shown based on Figure 30 The image 107 of the rear of the truck shown in FIG. White disparity pixels are located in the center of the second disparity image 150. In other words, the disparity of the disparity pixels in the center is not calculated in the second disparity image 150. The disparity of the disparity pixels surrounding the center is calculated in the second disparity image 150.

[0242] Figure 40 Second parallax image 150 shown includes parallax pixels 151, 152, 153, and 154. Parallax pixel 151 is used to calculate the second candidate height. The coordinates of parallax pixel 151 are associated with the second candidate height as candidate coordinates. Parallax pixels 152 and 153 are used to calculate the height of an object. Each of the coordinates of parallax pixel 152 and 153 is associated with the height of the object as a first coordinate. Parallax pixel 154 is used to calculate the first candidate height. The coordinates of parallax pixel 154 are associated with the first candidate height as a first coordinate.

[0243] and Figure 29 Similar to the parallax pixel 105 shown, the parallax pixel 151 is Figure 30 The upper image 107b corresponds to the image 107 of the rear of the truck shown. Figure 29 Similar to the parallax pixel 103 shown, the parallax pixel 154 is similar to the Figure 30 The lower image 107a corresponds to the image 107 of the back of the truck shown. Figure 40 In the structure shown, it is necessary to obtain the second candidate height based on the parallax pixel 151 as the height of the object, that is, the second candidate height based on the parallax pixel 151 and the first candidate height based on the parallax pixel 154. Figure 30 Height of truck shown.

[0244] Figure 41 The second parallax image 160 is shown based on Figure 42A parallax image is generated based on the first image 170 shown. First image 170 includes image 171 of the upper portion of another vehicle and image 172 corresponding to a street tree. The parallax generated by image 171 is approximately equal to the parallax generated by image 172. Image 172 includes partial image 172a and partial image 172b. Partial image 172a and vehicle image 171 are located in the center of first image 170 in the u direction. The u coordinate of partial image 172a is partially identical to the u coordinate of the other vehicle image 171. Partial image 172a is closer to the negative side of the v-axis than the other vehicle image 171. Partial image 172b is closer to the negative side of the u-axis than partial image 172a and image 171.

[0245] Figure 41 The second parallax image 160 shown includes parallax pixels 161, 162, and 163. Parallax pixels 161 are parallax pixels used to calculate the second candidate height. Figure 42 The height of the partial image 172a of the street tree shown is used as the second candidate height. The coordinates of the parallax pixel 161 are associated with the second candidate height as candidate coordinates. The parallax pixel 162 is a parallax pixel used to calculate the height of the object. Figure 42 The height of the partial image 172b of the street tree shown is used as the height of the object. The coordinates of the parallax pixel 162 are associated with the height of the object as candidate coordinates. The parallax pixel 163 is a parallax pixel used to calculate the first candidate height. Figure 42 The height of the vehicle image 171 is shown as the first candidate height. The coordinates of the parallax pixel 163 are associated with the first candidate height as the first coordinates.

[0246] exist Figure 41 In the structure shown, it is necessary to obtain the first candidate height based on the parallax pixel 163 and the second candidate height based on the parallax pixel 161 as the height of the object. Figure 42 The height of the image 171 of the vehicle is shown.

[0247] In the processing of steps S805 to S807 , the control unit 24 determines which of the first candidate height and the second candidate height to acquire as the height of the object based on the parallax associated with the two coordinates sandwiching the candidate coordinates.

[0248] Specifically, in the process of step S805, the control unit 24 scans the second parallax image along the u direction to determine whether there are two first coordinates sandwiching the candidate coordinates in the u direction. Figure 23As shown, a parallax substantially equal to the target parallax is associated with the first coordinate. That is, the process of step S805 can also be regarded as determining whether the parallax associated with the two coordinates sandwiching the candidate coordinate in the u direction is the target parallax.

[0249] For example, in Figure 40 In the u direction of the second parallax image 150 shown, the candidate coordinates of the parallax pixel 151 are sandwiched between the first coordinates of the parallax pixel 152 and the first coordinates of the parallax pixel 153. Figure 40 In this configuration, the control unit 24 determines that the first coordinates of the parallax pixel 152 and the first coordinates of the parallax pixel 153 exist between the candidate coordinates of the parallax pixel 151 .

[0250] For example, in Figure 41 In the u direction of the second parallax image 160 shown, the first coordinate of the parallax pixel 162 is located on the negative side of the u axis of the candidate coordinates of the parallax pixel 161. On the other hand, the first coordinate is not located on the positive side of the u axis of the candidate coordinates of the parallax pixel 161. Figure 41 In the illustrated configuration, the control unit 24 determines that there are no two first coordinates sandwiching the candidate coordinates of the parallax pixel 161 .

[0251] In step S805, if the control unit 24 determines that two first coordinates sandwiching the candidate coordinates exist in the u direction of the second parallax image (step S805: Yes), the process proceeds to step S806. On the other hand, if the control unit 24 determines that two first coordinates sandwiching the candidate coordinates do not exist in the u direction of the second parallax image (step S805: No), the process proceeds to step S807.

[0252] In the processing of step S806, the control unit 24 determines to obtain the second candidate height of the first candidate height and the second candidate height as the height of the object. Through the processing of steps S805 and S806, the control unit 24 obtains the second candidate height as the height of the object when it determines that the parallax corresponding to the two coordinates sandwiching the candidate coordinates is the object parallax. For example, the control unit 24 determines to obtain the second candidate height based on Figure 40 Of the first candidate height based on the coordinates of the parallax pixel 154 and the second candidate height based on the coordinates of the parallax pixel 151, the second candidate height based on the coordinates of the parallax pixel 151 is used as the height of the object. Figure 43 It is shown in Figure 40 The parallax pixels used to determine the height of the object in the second parallax image 150 are shown. Figure 43 In , the height of the object is calculated based on the coordinates of the disparity pixels marked with thin hatching. Figure 43As shown, the second candidate height based on the parallax pixel 151 and the first candidate height based on the parallax pixel 154 are obtained as the height of the object. Figure 30 Height of truck shown.

[0253] In the process of step S807, the control unit 24 determines to acquire the first candidate height of the first candidate height and the second candidate height as the height of the object. Through the processes of steps S805 and S807, the control unit 24 determines to acquire the first candidate height as the height of the object when the parallax corresponding to the two coordinates sandwiching the candidate coordinates is not determined to be the object parallax. For example, the control unit 24 determines to acquire the first candidate height based on Figure 41 Of the first candidate height based on the coordinates of the parallax pixel 163 and the second candidate height based on the coordinates of the parallax pixel 161, the first candidate height based on the coordinates of the parallax pixel 163 is used as the height of the object. Figure 44 It is shown in Figure 41 The parallax pixels used to determine the height of the object in the second parallax image 160 are shown. Figure 44 In , the height of the object is calculated based on the coordinates of the disparity pixels marked with thin hatching. Figure 44 As shown, the first candidate height based on the parallax pixel 163 and the second candidate height based on the parallax pixel 161 are obtained as the height of the object. Figure 42 The height of the image 171 of the vehicle is shown.

[0254] After executing steps S806 and S807, the control unit 24 enters Figure 5 The process of step S108 is shown.

[0255] In the process of step S108, the control unit 24 performs a process of determining the representative parallax. Figure 45 The details of the process of step S108 are shown in the flowchart shown.

[0256] In the process of step S901, the control unit 24 performs Figure 31 The same or similar processing as step S701 is performed to generate a UD graph. Figure 31 When a UD diagram is generated in the process of step S701 shown in FIG. Figure 31 The UD map generated in the process of step S701 is shown.

[0257] In the process of step S902, the control unit 24 selects the first parallax and the u coordinates of the UD map. Figure 37The control unit 24 obtains the representative disparity from the second disparity to which the restoration flag is added in the process of step S804. Figure 31 The process of step S706 shown above obtains a representative disparity from the first disparity to which the parallel object flag is not attached and the second disparity to which the restoration flag is attached.

[0258] When the distance from the stereo camera 10 to the object is close, the object may occupy more pixels in the stereo image than when the distance from the stereo camera 10 to the object is far. Furthermore, the farther an object is from the stereo camera 10, the more it is affected by noise and other factors, which can reduce the accuracy of the parallax detection corresponding to the object. In other words, the closer an object is to the stereo camera 10, the higher the accuracy of the parallax detection corresponding to the object.

[0259] Therefore, in the process of step S902, the control unit 24 obtains the parallax corresponding to the object close to the stereo camera 10, that is, the largest parallax, from the first parallax and the second parallax to which the restoration flag is added, as the representative parallax. Furthermore, the control unit 24 may obtain the parallax corresponding to the object close to the stereo camera 10, that is, the object whose height is greater than the minimum value (e.g., 50 cm) of the objects to be detected by the object detection device 20, as the representative parallax.

[0260] exist Figure 46 An example of the representative parallax obtained corresponding to the u direction is shown in FIG. Figure 46 In the example shown, the representative disparity with respect to each u coordinate varies greatly. If the representative disparity with respect to each u coordinate varies greatly, the amount of computation required for the grouping process in step S109 may increase.

[0261] Therefore, in the processing of step S903, the control unit 24 can average the representative disparities within a specified range. By averaging the representative disparities within a specified range, the amount of computation required for the grouping processing in the processing of step S109 can be reduced. The specified range can be appropriately set in consideration of the computational load of the processing of step S109 described later. In performing the averaging, the control unit 24 can obtain the representative disparity of the central value of the specified range. The control unit 24 can calculate the average value of the representative disparities by removing the representative disparities that are offset by a specified ratio (e.g., 5%) from the extracted central value within the specified range. Figure 47 An example of the averaged representative parallax corresponding to the u direction is shown in FIG. After executing the process of step S903, the control unit 24 enters the Figure 5 The process of step S109 is shown.

[0262] The control unit 24 can execute Figure 5After the processing of steps S104 to S108 shown in the figure, that is, the third processing, for each predetermined range of horizontal coordinates (u coordinates) including one or more coordinates, the parallax d is represented. e The height information of the object is associated with the object and stored in the memory 23. Figure 48 As shown, the plurality of representative disparities d stored in the memory 23 e It can be expressed as the distribution of a point group in a two-dimensional space (ud coordinate space) with the u coordinate and the parallax d as the horizontal axis and the vertical axis respectively.

[0263] In the process of step S109, the control unit 24 calculates the representative parallax d of the ud coordinate space by e The information is transformed into a coordinate system of the real space composed of xz coordinates, and the representative parallax d is extracted. e , to perform the process of detecting the object (the fourth process). Figure 49 as well as Figure 50 , an example of the processing of step S109 is described.

[0264] Figure 49 This is an example of a structure in real space. Figure 49 , a moving body 30 equipped with the object detection system 1 and another vehicle 42A are shown traveling on a road surface 41A. Figure 49 In FIG, the moving object 30 is a vehicle.

[0265] The control unit 24 of the object detection device 20 mounted on the moving body 30 Figure 48 The multiple representative disparities d in the ud coordinate space shown e Transformed into Figure 50 The point group of the actual space (xz coordinate space) shown in . Figure 50 In the ud coordinate space, the representative disparity d is represented e Each point is displayed as a point in the xz coordinate space. The control unit 24 extracts a set of point groups based on the distribution of the point group. The control unit 24 aggregates a plurality of points that are close to each other according to a predetermined condition and extracts the set of point groups. The set of point groups represents the representative parallax d e A collection (group) of .

[0266] When an object has a surface parallel to the baseline length direction of the stereo camera 10, the point group is arranged along the x direction in the xz coordinate space. When there is a set 180 of point groups arranged along the x direction in the xz coordinate space, the control unit 24 can recognize it as an object. Figure 50 In the point group, the concentration of 180 and Figure 49 The rear side of the vehicle body of the other vehicle 42A shown corresponds.

[0267] When the object is a parallel object, the point group is arranged along the z direction in the xz coordinate space. When there is a set 181 of point groups arranged along the z direction in the xz coordinate space, the control unit 24 can recognize it as a parallel object. As mentioned above, examples of parallel objects include guardrails and roadside buildings such as sound insulation walls on highways, or Figure 49 The side of the other vehicle 42A shown, etc. The set 181 of point groups arranged in the z direction in the xz coordinate space corresponds to an object arranged parallel to the direction of travel of the moving body 30, or a surface parallel to the direction of travel of the moving body 30. The control unit 24 can exclude the set 181 of point groups arranged in the z direction from the objects to be detected. The parallax corresponding to the parallel objects detected in the above clustering process is regarded as Figure 45 The representative parallax d in the process of step S902 is shown as e However, in clustering processing, parallaxes corresponding to parallel objects may not be fully detected. In this case, parallaxes corresponding to parallel objects may exist in the xz coordinate space, such as point cluster 181. Even if parallaxes corresponding to parallel objects exist in the xz coordinate space, such as point cluster 181, parallaxes corresponding to parallel objects can be excluded from the object detection processing in step S109.

[0268] The control unit 24 can detect the width of the object based on the width of the set 180 of point groups recognized as the object arranged in the x direction. Figure 45 The representative disparity d obtained by the process shown e Therefore, the control unit 24 can identify the position, horizontal width, and height of the identified object in the xz coordinate space.

[0269] In the process of step S110, the control unit 24 can output the information of the position, lateral width and height of the object recognized by the process of step S109 to other devices in the mobile body 30 through the output unit 22. For example, the control unit 24 can output this information to a display device in the mobile body 30. Figure 51 As shown, the display device in the mobile object 30 can display a detection frame 182 surrounding the image corresponding to the other vehicle in the image of the first camera 11 or the second camera 12 based on the information obtained from the object detection device 20. The detection frame 182 indicates the position of the detected object and the range occupied by the object in the image.

[0270] As described above, the object detection device 20 of the present invention can achieve high processing speed and high accuracy in object detection. That is, the object detection device 20 and the object detection method of the present invention can improve object detection performance. Furthermore, the object detection device 20 does not limit detection targets to specific types of objects. The object detection device 20 can detect all objects on the road surface. The control unit 24 of the object detection device 20 can perform the first, second, third, and fourth processes without using information from images other than the first parallax image captured by the stereo camera 10. Therefore, in addition to processing the first and second parallax images, the object detection device 20 can also avoid performing processing to identify objects based solely on the captured images. Therefore, the object detection device 20 of the present invention can reduce the processing load on the control unit 24 during object recognition. This does not preclude the object detection device 20 of the present invention from being combined with image processing of images directly obtained from the first camera 11 or the second camera 12. The object detection device 20 can also be combined with image processing techniques such as template matching.

[0271] In the above description of the processing executed by the control unit 24, to facilitate understanding of the present invention, the processing includes determinations and operations using various images. The processing using these images does not necessarily include actual image rendering. Processing substantially identical to the processing using these images is executed through information processing within the control unit 24.

[0272] The embodiments of the present invention have been described based on the accompanying drawings and embodiments, but it should be noted that it is easy for those skilled in the art to make various deformations or modifications based on the present invention. Therefore, it should be noted that these deformations or modifications are included in the scope of the present invention. For example, the functions included in each component or each step can be reconfigured in a logically non-contradictory manner, and multiple components or steps can be combined into one or divided. The embodiments of the present invention have been described with the device as the center, but the embodiments of the present invention can also be implemented as a method including the steps performed by each component of the device. The embodiments of the present invention can be implemented as a method executed by a processor possessed by the device, a program, or a storage medium having a program recorded thereon. It should be understood that these contents are also included in the scope of the present invention.

[0273] In the present invention, the descriptions of "first" and "second" are identifiers used to distinguish the structures. The structures distinguished by the descriptions of "first" and "second" in the present invention can exchange the numbers in the structures. For example, the first lens can exchange the "first" and "second" as identifiers with the second lens. The exchange of identifiers is carried out simultaneously. The structures can also be distinguished after the identifiers are exchanged. The identifiers can be deleted. The structures with deleted identifiers are distinguished by figure marks. The descriptions of identifiers such as "first" and "second" in the present invention should not be used to interpret the order of the structures, nor can they be used as a basis for the existence of identifiers with smaller numbers.

[0274] In the present invention, the x-direction, y-direction, and z-direction are provided for ease of description and may be interchanged. The structure of the present invention is described using an orthogonal coordinate system with the x-direction, y-direction, and z-direction as the directions of the respective axes. The positional relationship of the various structures of the present invention is not limited to being orthogonal. The u-coordinate and v-coordinate representing the coordinates of the image are provided for ease of description and may be interchanged. The origin and direction of the u-coordinate and v-coordinate are not limited to the content of the present invention.

[0275] In the above embodiment, the first camera 11 and the second camera 12 of the stereo camera 10 are arranged in the x direction. The configuration of the first camera 11 and the second camera 12 is not limited to this. The first camera 11 and the second camera 12 can be arranged in a direction perpendicular to the road surface (y direction) or in a direction inclined relative to the road surface 41A. The number of cameras constituting the stereo camera 10 is not limited to two. The stereo camera 10 can include more than three cameras. For example, a total of four cameras, namely, two cameras arranged in a horizontal direction on the road surface and two cameras arranged in a vertical direction, can be used to obtain distance information with higher accuracy.

[0276] In the above embodiment, the stereo camera 10 and the object detection device 20 are mounted on the mobile object 30. However, the stereo camera 10 and the object detection device 20 are not limited to being mounted on the mobile object 30. For example, the stereo camera 10 and the object detection device 20 may also be mounted on a roadside device installed at an intersection, etc., to capture images including the road surface. For example, the roadside device can provide information such as detecting a first vehicle approaching from one side of the intersecting roads at the intersection and notifying a second vehicle approaching on the other side of the road of the first vehicle's approach.

[0277] Description of reference numerals:

[0278] 1.1A Object Detection System

[0279] 10 Stereo Camera

[0280] 11 First Camera

[0281] 12 Second Camera

[0282] 20 Object Detection Device

[0283] 21 Acquisition Department

[0284] 22 Output

[0285] 23 Memory

[0286] 24 Control Department

[0287] 25 Generator

[0288] 30 Mobile

[0289] 40 First parallax image

[0290] 41, 42, 43, 44, 71, 72, 73, 74, 75, 76, 81, 82, 101 parallax images

[0291] Road 41A

[0292] 42A Vehicle

[0293] 42a, 83, 84, 85, 102, 103, 104, 105, 151, 152, 153, 154, 161, 162, 163 parallax pixels

[0294] 45 frame line

[0295] 51 graphics

[0296] 52 First Line

[0297] 53 Approximate starting point

[0298] 54 Alternative straight lines

[0299] 55 Second straight line

[0300] 60, 70, 80, 100, 150, 160 Second parallax image

[0301] 61 Partial area

[0302] 61-1, 61 parts of the area

[0303] 62 Area

[0304] 90, 106, 120, 170 First image

[0305] Images 91, 92, 107, 121, 122, 123, 124, 125, 171, 172

[0306] Part of 101a

[0307] 107a Lower image

[0308] 107b Upper image

[0309] 110, 140 UD images

[0310] 110VP, 120VP Vanishing Point

[0311] Point groups 111, 112, 113, 114, 115, 180, and 181

[0312] 111L, 115L straight line

[0313] 131, 132, 133, 134 points

[0314] 131L, 132L, 133L, 134L, 135L curves

[0315] 141, 142 First parallax point

[0316] 143 Second Parallax Point

[0317] Partial images 172a and 172b

[0318] 182 detection frame

Claims

1. An object detection device, wherein: A processor is provided, the processor being configured to search for coordinates corresponding to a parallax substantially equal to an object parallax satisfying a predetermined condition along a second direction of a disparity map corresponding to a two-dimensional coordinate composed of a first direction corresponding to a horizontal direction of a captured image generated by a stereo camera capturing a road surface and a second direction intersecting the first direction, and to update the coordinates to the first coordinates, and calculate a height of an object corresponding to the object parallax based on the first coordinates. The processor is further configured to: When a second coordinate associated with a parallax substantially equal to the object parallax exists within a predetermined interval from the first coordinate, updating the second coordinate to the first coordinate; When there are more than a prescribed number of coordinates corresponding to the parallax smaller than the object parallax between the first coordinate and the second coordinate, the second coordinate is not updated to the first coordinate, The processor is further configured to: dividing the disparity map into a plurality of partial regions along the first direction of the disparity map, and generating a distribution indicating the frequency of disparity for each of the plurality of partial regions, extracting the disparity whose frequency exceeds a predetermined threshold as the target disparity satisfying the predetermined condition, The parallax substantially equal to the object parallax includes the same parallax as the object parallax and the substantially same parallax as the object parallax, and the parallax substantially equal to the object parallax is defined as the parallax having a difference from the object parallax within a range of ±10%.

2. The object detection device according to claim 1, wherein: The processor is configured as follows: Along the direction corresponding to the upward direction from the road surface among the directions included in the second direction of the disparity map, the coordinates corresponding to the object parallax are searched from the coordinates of the road surface corresponding to the object parallax and updated to the first coordinates, and the height of the object is calculated by subtracting the coordinates of the road surface from the first coordinates.

3. The object detection device according to claim 2, wherein: The processor is configured as follows: If there are no coordinates corresponding to a parallax substantially equal to the object parallax within a predetermined range from the coordinates of the road surface along a direction corresponding to an upward direction from the road surface, the height of the object is not calculated.

4. The object detection device according to claim 3, wherein: The predetermined range is set based on the height of the object floating from the road surface from the road surface.

5. The object detection device according to any one of claims 1 to 4, wherein: The predetermined interval is set based on the height of an object to be detected by the object detection device.

6. The object detection device according to claim 1, wherein: The predetermined interval is set based on the height of an object to be detected by the object detection device.

7. The object detection device according to any one of claims 1 to 4, wherein: The processor is configured to execute the following processing as a process of generating the disparity map: A first process of estimating a shape of a road surface in real space based on a first disparity map, wherein the first disparity map is generated based on output from the stereo camera and is a map in which two-dimensional coordinates formed by the first direction and the second direction are associated with disparities obtained from the output of the stereo camera; as well as The second process generates a second disparity map as the disparity map by removing disparities corresponding to a range of heights below a predetermined height from the road surface in real space from the first disparity map based on the estimated shape of the road surface.

8. The object detection device according to claim 1, wherein: The processor is configured to execute the following processing as a process of generating the disparity map: A first process of estimating a shape of a road surface in real space based on a first disparity map, wherein the first disparity map is generated based on output from the stereo camera and is a map in which two-dimensional coordinates formed by the first direction and the second direction are associated with disparities obtained from the output of the stereo camera; as well as The second process generates a second disparity map as the disparity map by removing disparities corresponding to a range of heights below a predetermined height from the road surface in real space from the first disparity map based on the estimated shape of the road surface.

9. The object detection device according to claim 5, wherein: The processor is configured to execute the following processing as a process of generating the disparity map: A first process of estimating a shape of a road surface in real space based on a first disparity map, wherein the first disparity map is generated based on output from the stereo camera and is a map in which two-dimensional coordinates formed by the first direction and the second direction are associated with disparities obtained from the output of the stereo camera; as well as The second process generates a second disparity map as the disparity map by removing disparities corresponding to a range of heights below a predetermined height from the road surface in real space from the first disparity map based on the estimated shape of the road surface.

10. The object detection device according to claim 6, wherein: The processor is configured to execute the following processing as a process of generating the disparity map: A first process of estimating a shape of a road surface in real space based on a first disparity map, wherein the first disparity map is generated based on output from the stereo camera and is a map in which two-dimensional coordinates formed by the first direction and the second direction are associated with disparities obtained from the output of the stereo camera; as well as The second process generates a second disparity map as the disparity map by removing disparities corresponding to a range of heights below a predetermined height from the road surface in real space from the first disparity map based on the estimated shape of the road surface.

11. An object detection system, wherein: have: Stereo cameras capture multiple images with mutual parallax; as well as An object detection device comprising at least one processor, The at least one processor is configured to: A method for determining a disparity map corresponding to a disparity obtained from a captured image captured by the stereo camera, wherein two-dimensional coordinates formed by a first direction corresponding to a horizontal direction of the captured image generated by the stereo camera and a second direction intersecting the first direction are associated with the captured image, searches for coordinates corresponding to a disparity substantially equal to an object disparity satisfying a specified condition, updates the coordinates to the first coordinates, and calculates a height of an object corresponding to the object disparity based on the first coordinates. When a second coordinate associated with a parallax substantially equal to the object parallax exists within a predetermined interval from the first coordinate, updating the second coordinate to the first coordinate; When there are more than a prescribed number of coordinates corresponding to the parallax smaller than the object parallax between the first coordinate and the second coordinate, the second coordinate is not updated to the first coordinate, The at least one processor is further configured to: dividing the disparity map into a plurality of partial regions along the first direction of the disparity map, and generating a distribution indicating the frequency of disparity for each of the plurality of partial regions, extracting the disparity whose frequency exceeds a predetermined threshold as the target disparity satisfying the predetermined condition, The parallax substantially equal to the object parallax includes the same parallax as the object parallax and the substantially same parallax as the object parallax, and the parallax substantially equal to the object parallax is defined as the parallax having a difference from the object parallax within a range of ±10%.

12. A mobile object, wherein: An object detection system is provided, the object detection system comprising: a stereo camera for capturing a plurality of images having parallax with respect to each other; and an object detection device comprising at least one processor, The at least one processor is configured to: A method for determining a disparity map corresponding to a disparity obtained from a captured image captured by the stereo camera, wherein two-dimensional coordinates formed by a first direction corresponding to a horizontal direction of the captured image generated by the stereo camera and a second direction intersecting the first direction are associated with the captured image, searches for coordinates corresponding to a disparity substantially equal to an object disparity satisfying a specified condition, updates the coordinates to the first coordinates, and calculates a height of an object corresponding to the object disparity based on the first coordinates. When a second coordinate associated with a parallax substantially equal to the object parallax exists within a predetermined interval from the first coordinate, updating the second coordinate to the first coordinate; When there are more than a prescribed number of coordinates corresponding to the parallax smaller than the object parallax between the first coordinate and the second coordinate, the second coordinate is not updated to the first coordinate, The at least one processor is further configured to: dividing the disparity map into a plurality of partial regions along the first direction of the disparity map, and generating a distribution indicating the frequency of disparity for each of the plurality of partial regions, extracting the disparity whose frequency exceeds a predetermined threshold as the target disparity satisfying the predetermined condition, The parallax substantially equal to the object parallax includes the same parallax as the object parallax and the substantially same parallax as the object parallax, and the parallax substantially equal to the object parallax is defined as the parallax having a difference from the object parallax within a range of ±10%.

13. A method for object detection, wherein: The steps include: A method for determining a disparity map corresponding to a disparity obtained from a captured image captured by a stereo camera, wherein two-dimensional coordinates formed by a first direction corresponding to a horizontal direction of the captured image and a second direction intersecting the first direction are associated with the captured image, searches for coordinates corresponding to a disparity substantially equal to an object disparity satisfying a specified condition, updates the coordinates to the first coordinates, and calculates a height of an object corresponding to the object disparity based on the first coordinates. The step of calculating the height of the object comprises: When a second coordinate associated with a parallax substantially equal to the object parallax exists within a predetermined interval from the first coordinate, updating the second coordinate to the first coordinate; When there are more than a prescribed number of coordinates corresponding to the parallax smaller than the object parallax between the first coordinate and the second coordinate, the second coordinate is not updated to the first coordinate, The step of calculating the height of the object further comprises: dividing the disparity map into a plurality of partial regions along the first direction of the disparity map, and generating a distribution indicating the frequency of disparity for each of the plurality of partial regions, extracting the disparity whose frequency exceeds a predetermined threshold as the target disparity satisfying the predetermined condition, The parallax substantially equal to the object parallax includes the same parallax as the object parallax and the substantially same parallax as the object parallax, and the parallax substantially equal to the object parallax is defined as the parallax having a difference from the object parallax within a range of ±10%.

Images