Object detection device, object detection system, moving object, and object detection method

Abstract

The object detection device includes a processor. The processor is configured to detect a straight line of a specified length by applying Hough transform to the coordinate points of the UD map, and detect the object disparity corresponding to the detected straight line of the specified length as the disparity corresponding to the object parallel to the direction of travel of the stereo camera. The UD map is a map in which the object disparity that satisfies a specified condition among the disparities obtained from the captured image is associated with coordinate points of two-dimensional coordinates consisting of a first direction and a direction corresponding to the magnitude of the disparity. The first direction corresponds to the horizontal direction of the captured image generated by the stereo camera capturing the road surface. The processor is configured to convert each coordinate point associated with the object disparity and a straight line of a specified range based on the vanishing point into the Hough space in the Hough transform.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Japanese Patent Application No. 2019-170900, filed on September 19, 2019, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to an object detection device, an object detection system, a mobile object, and an object detection method.

[0004] In recent years, object detection devices using stereo cameras have been installed on mobile objects such as automobiles. Such object detection devices acquire multiple images from the stereo camera and detect objects that may be obstacles based on the acquired multiple images (for example, see Patent Document 1).

[0005] Prior art literature

[0006] Patent Literature

[0007] Special Publication No. 5-265547

[0008] An object detection device according to one embodiment of the present disclosure includes a processor. The processor is configured to detect a straight line of a specified length by applying a Hough transform to the coordinate points of a U-Disparity (UD) map, and detect the object disparity corresponding to the detected straight line of the specified length as the disparity corresponding to an object parallel to the direction of travel of the stereo camera. The UD map is a map in which the object disparity that satisfies a specified condition among the disparities obtained from the captured image is associated with coordinate points of two-dimensional coordinates consisting of a first direction and a direction corresponding to the magnitude of the disparity. The first direction corresponds to the horizontal direction of the captured image generated by capturing a road surface by a stereo camera. The processor is configured to convert each coordinate point associated with the object disparity and a straight line of a specified range based on a vanishing point into a Hough space in the Hough transform.

[0009] An object detection system according to one embodiment of the present disclosure comprises: a stereo camera that captures a plurality of images having parallaxes with respect to each other; and an object detection device comprising at least one processor. The processor is configured to detect a straight line of a specified length by applying a Hough transform to the coordinate points of the UD map, and detect the object parallax corresponding to the detected straight line of the specified length as the parallax corresponding to an object parallel to the direction of travel of the stereo camera. The UD map is a map in which the object parallax that satisfies a specified condition among the parallaxes obtained from the captured image is associated with a coordinate point of a two-dimensional coordinate consisting of a first direction and a direction corresponding to the magnitude of the parallax. The first direction corresponds to the horizontal direction of the captured image generated by the stereo camera capturing the road surface. The processor is configured to convert each coordinate point associated with the object parallax and a straight line within a specified range based on a vanishing point into a Hough space in the Hough transform.

[0010] A mobile body according to one embodiment of the present disclosure is provided with an object detection system. The object detection system comprises: a stereo camera that captures a plurality of images having parallaxes with respect to each other; and an object detection device comprising at least one processor. The processor is configured to detect a straight line of a predetermined length by applying a Hough transform to the coordinate points of the UD map, and detect the object parallax corresponding to the detected straight line of the predetermined length as the parallax corresponding to an object parallel to the direction of travel of the stereo camera. The UD map is a map in which the object parallax that satisfies a predetermined condition among the parallaxes obtained from the captured image is associated with a coordinate point of a two-dimensional coordinate consisting of a first direction and a direction corresponding to the magnitude of the parallax. The first direction corresponds to the horizontal direction of the captured image generated by capturing a road surface by the stereo camera. The processor is configured to convert each coordinate point associated with the object parallax and a straight line within a predetermined range based on a vanishing point into a Hough space in the Hough transform.

[0011] An object detection method according to an embodiment of the present disclosure includes detecting the object disparity corresponding to the detected straight line of the prescribed length as the disparity corresponding to an object parallel to the direction of travel of the stereo camera. In the disparity detected as corresponding to the parallel object, the straight line of the prescribed length is detected by applying the Hough transform to the coordinate points of the UD map. The UD map is a map in which the object disparity that satisfies prescribed conditions among the disparities obtained from the captured image is associated with coordinate points of two-dimensional coordinates consisting of a first direction and a direction corresponding to the magnitude of the disparity. The first direction corresponds to the horizontal direction of the captured image generated by the stereo camera capturing the road surface. Detecting the disparity corresponding to the parallel object includes converting the coordinate points associated with the object disparity and the straight line within a prescribed range based on the vanishing point into the Hough space in the Hough transform. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a block diagram showing a schematic configuration of an object detection system according to one embodiment of the present disclosure.

[0013] Figure 2 is a schematic diagram showing the installation Figure 1 Side view of a moving body of an object detection system.

[0014] Figure 3 is a schematic diagram showing the installation Figure 1 Front view of a moving body of an object detection system.

[0015] Figure 4 This is a block diagram showing a schematic configuration of an object detection system according to another embodiment of the present disclosure.

[0016] Figure 5 It shows Figure 1 A flowchart of an example of processing performed by an object detection device.

[0017] Figure 6 A diagram illustrating an example of a first parallax image acquired or generated by the object detection device.

[0018] Figure 7 This is a flowchart showing an example of a process for estimating a road surface shape.

[0019] Figure 8 This is a flowchart showing an example of a process of extracting a road surface candidate parallax from a first parallax image.

[0020] Figure 9 A diagram illustrating the positional relationship between the road surface and the stereo camera.

[0021] Figure 10 3 is a diagram illustrating the process of extracting candidate road surface disparities.

[0022] Figure 11 FIG. 1 is a diagram showing a range on a road surface obtained by converting a parallax histogram.

[0023] Figure 12 is the road parallax d r A dv correlation diagram showing an example of the relationship between the vertical coordinate (v coordinate).

[0024] Figure 13 This is a diagram explaining a method of detecting whether an object other than road surface parallax is included.

[0025] Figure 14 is the parallax d of the road surface through the straight line r Flowchart of the process of approximating the relationship between the vertical coordinate (v coordinate) and the image.

[0026] Figure 15 This is an illustration of the road parallax d generated by the first straight line. r An approximate diagram of .

[0027] Figure 16 This is a diagram explaining the method of determining the second straight line.

[0028] Figure 17 is a diagram showing how to approximate the road parallax d using a straight line. r A diagram showing an example of the result of the relationship between and the vertical coordinate (v coordinate) of the image.

[0029] Figure 18 A diagram showing an example of a second parallax image.

[0030] Figure 19 It is the reference image of the second parallax image.

[0031] Figure 20 This is a flowchart showing an example of a process for detecting the first parallax and the second parallax.

[0032] Figure 21 Is to combine some areas with Figure 18 The diagram shows a second parallax image overlap representation.

[0033] Figure 22 : is a diagram showing an example of a parallax histogram.

[0034] Figure 23 This is a flowchart (part 1) showing an example of object height calculation processing.

[0035] Figure 24 This is a flowchart (part 2) showing an example of object height calculation processing.

[0036] Figure 25 This is a flowchart (part 3) showing an example of object height calculation processing.

[0037] Figure 26 A diagram showing an example of a second parallax image.

[0038] Figure 27 A diagram showing an example of a second parallax image.

[0039] Figure 28 It shows Figure 27 The first image corresponding to the second parallax image shown.

[0040] Figure 29 A diagram showing an example of a second parallax image.

[0041] Figure 30 yes Figure 29 The first image corresponding to the second parallax image shown.

[0042] Figure 31 This is a flowchart showing an example of parallel object detection processing.

[0043] Figure 32 This is a diagram showing an example of a UD diagram.

[0044] Figure 33 yes Figure 32 The first image corresponding to the UD image shown.

[0045] Figure 34 This is a diagram showing a UD diagram from which a point group substantially parallel to the u direction has been removed.

[0046] Figure 35 Figure 1 illustrating the Hough transform.

[0047] Figure 36 Figure 2 illustrating the Hough transform.

[0048] Figure 37 This is a flowchart showing an example of the restoration process.

[0049] Figure 38 This is a diagram showing an example of a UD diagram.

[0050] Figure 39 This is a diagram showing an example of a UD map to which a restoration flag is added.

[0051] Figure 40 A diagram showing an example of a second parallax image.

[0052] Figure 41 A diagram showing an example of a second parallax image.

[0053] Figure 42 yes Figure 41 The diagram shows the first image corresponding to the second parallax image.

[0054] Figure 43 It is shown in Figure 40 A diagram showing disparity pixels used for object height determination in the second disparity image is shown.

[0055] Figure 44 It is shown in Figure 41 A diagram showing disparity pixels used for object height determination in the second disparity image is shown.

[0056] Figure 45 This is a flowchart showing an example of a process of determining a representative parallax.

[0057] Figure 46 An example of the representative parallax corresponding to the acquired u direction is shown.

[0058] Figure 47An example of the averaged representative parallax corresponding to the u direction is shown.

[0059] Figure 48 This is a diagram showing an example of distribution in a UD map showing a point group representing parallax.

[0060] Figure 49 This is a diagram of the road surface viewed from the height direction (y direction).

[0061] Figure 50 3 is a diagram showing the point group converted into a point group on the xz plane representing the real space representing the parallax.

[0062] Figure 51 This is a diagram showing an example of a method for outputting object detection results. DETAILED DESCRIPTION

[0063] In existing object detection devices, it is necessary to improve the performance of object detection. The object detection device, object detection system, mobile object and object detection method disclosed in the present invention can improve the performance of object detection.

[0064] Hereinafter, an embodiment of the present disclosure will be described with reference to the accompanying drawings. In the following drawings, the same or similar components are represented by the same reference numerals. It should be noted that the figures used in the following description are schematic. The dimensions and proportions in the drawings are not necessarily consistent with the actual dimensions and proportions. The figures showing the captured images and parallax images captured by the camera include figures produced for the convenience of explanation. These images are different from the images actually captured or processed. In addition, in the following description, the "subject" is the object captured by the camera. The "subject" includes objects, roads, and the sky, etc. "Objects" have a specific position and size in space. "Objects" are also called "three-dimensional objects."

[0065] like Figure 1 As shown, the object detection system 1 includes a stereo camera 10 and an object detection device 20. The stereo camera 10 and the object detection device 20 can communicate through wired or wireless communication. The stereo camera 10 and the object detection device 20 can communicate through a network. The network may include, for example, a wired or wireless LAN (Local Area Network) or CAN (Controller Area Network). The stereo camera 10 and the object detection device 20 can be housed in the same housing and constructed as one body. The stereo camera 10 and the object detection device 20 can be constructed to be located in a moving body 30 described later, and can communicate with an ECU (ElecTronic Control Unit) in the moving body 30.

[0066] In the present disclosure, a "stereo camera" refers to a plurality of cameras that have parallax with each other and cooperate with each other. A stereo camera includes at least two or more cameras. In a stereo camera, a plurality of cameras can work together to shoot an object from multiple directions. A stereo camera may be a machine that includes a plurality of cameras in one housing. A stereo camera may be a machine that includes two or more cameras that are independent of each other and separated from each other. A stereo camera is not limited to a plurality of cameras that are independent of each other. In the present disclosure, for example, a camera having an optical mechanism that guides light incident on two separate places to a light receiving element may be used as a stereo camera. In the present disclosure, a plurality of images obtained by shooting the same subject from different viewpoints may be referred to as "stereo images."

[0067] like Figure 1 As shown, the stereo camera 10 includes a first camera 11 and a second camera 12. The first camera 11 and the second camera 12 respectively include an optical system and a shooting element that specifies an optical axis OX. The first camera 11 and the second camera 12 respectively have different optical axes OX. In this embodiment, only a single figure mark OX is used to uniformly represent the optical axes OX of both the first camera 11 and the second camera 12. The shooting element includes a CCD image sensor (Charge-Coupled Device Image Sensor) and a CMOS image sensor (Complementary MOS Image Sensor). The shooting elements respectively provided by the first camera 11 and the second camera 12 can exist in the same plane perpendicular to the optical axis OX of each camera. The first camera 11 and the second camera 12 generate an image signal representing an image imaged by the shooting element. In addition, the first camera 11 and the second camera 12 can perform arbitrary processing such as distortion correction, brightness adjustment, contrast adjustment and gamma correction on the captured image.

[0068] The optical axes OX of the first camera 11 and the second camera 12 are oriented in directions that allow them to capture the same subject. The optical axes OX and positions of the first camera 11 and the second camera 12 are determined so that the captured images contain at least the same subject. The optical axes OX of the first camera 11 and the second camera 12 are oriented parallel to each other. This parallelism is not necessarily strictly parallel; deviations in assembly and installation, as well as deviations over time, are permitted. The optical axes OX of the first camera 11 and the second camera 12 are not necessarily parallel and may be oriented in different directions. Even when the optical axes OX of the first camera 11 and the second camera 12 are not parallel to each other, a stereoscopic image can be generated by converting the images within the stereo camera 10 or the object detection device 20. The distance between the optical center of the first camera 11 and the optical center of the second camera 12 of the stereo camera 10 is called the baseline length. The baseline length is equivalent to the center-to-center distance between the optical centers of the first camera 11 and the second camera 12. The baseline length direction is the direction connecting the optical centers of the first camera 11 and the second camera 12 of the stereo camera 10.

[0069] The first camera 11 and the second camera 12 are positioned separately in a direction intersecting the optical axis OX. In one embodiment of multiple embodiments, the first camera 11 and the second camera 12 are positioned in the left-right direction. The first camera 11 is located on the left side of the second camera 12 when facing forward. The second camera 12 is located on the right side of the first camera 11 when facing forward. The first camera 11 and the second camera 12 capture the subject at a specified frame rate (for example, 30fps). Since the positions of the first camera 11 and the second camera 12 are different, the positions of the subjects corresponding to each other in the two images captured by each camera are different. The first camera 11 captures the first image. The second camera 12 captures the second image. The first image and the second image are stereo images captured from different viewpoints.

[0070] like Figure 2 and Figure 3 As shown, Figure 1 The object detection system 1 is installed on the moving body 30. Figure 2 As shown, the first camera 11 and the second camera 12 are arranged so that the optical axis OX of each optical system of the first camera 11 and the second camera 12 is substantially parallel to the front of the moving object 30 so as to be able to capture images of the front of the moving object 30 .

[0071] The mobile body 30 of the present disclosure travels on a travel path including a road, a track, etc. The surface of the travel path on which the mobile body 30 travels is also referred to as a "road surface."

[0072] In this disclosure, the direction of movement of the moving object 30 when moving straight ahead is also referred to as "forward" or "positive z-axis direction." The direction opposite to the forward direction is also referred to as "rearward" or "negative z-axis direction." In the absence of a distinction between the positive z-axis and the negative z-axis, they are collectively referred to as the "z-direction." The left and right directions are defined with reference to the forward direction of the moving object 30. The z-direction is also referred to as the "depth direction."

[0073] In this disclosure, the direction perpendicular to the z-direction and extending from the left to the right is also referred to as the "positive x-axis direction." The direction perpendicular to the z-direction and extending from the right to the left is also referred to as the "negative x-axis direction." In the absence of a distinction between the positive and negative x-axis directions, it is also referred to as the "x-direction." The x-direction may coincide with the baseline length direction. The x-direction is also referred to as the "horizontal direction."

[0074] In this disclosure, the direction perpendicular to the road surface near the moving object 30 and pointing upward from the road surface is also referred to as the "height direction" or the "positive y-axis direction." The direction opposite to the height direction is also referred to as the "negative y-axis direction." In situations where the positive and negative y-axis directions are not specifically distinguished, they are collectively referred to as the "y-direction." The y-direction may be orthogonal to the x-direction and the z-direction. The y-direction is also referred to as the "vertical direction."

[0075] The "mobile body" in the present disclosure may include, for example, vehicles and aircraft. Vehicles may include, for example, automobiles, industrial vehicles, railway vehicles, daily life vehicles, fixed-wing aircraft traveling on runways, etc. Automobiles may include, for example, passenger cars, trucks, buses, motorcycles, and trolleybuses, etc. Industrial vehicles may include, for example, industrial vehicles used in agriculture and construction. Industrial vehicles may include, for example, forklifts, golf carts, etc. Industrial vehicles used in agriculture may include, for example, tractors, tillers, transporters, harvesters, combine harvesters, lawn mowers, etc. Industrial vehicles used in construction may include, for example, bulldozers, scrapers, excavators, cranes, dump trucks, road rollers, etc. Vehicles may include manually driven vehicles. The classification of vehicles is not limited to the above examples. For example, automobiles may include industrial vehicles that can travel on roads. The same vehicle may be included in multiple categories. Aircraft may include, for example, fixed-wing aircraft, rotary-wing aircraft, etc.

[0076] The first camera 11 and the second camera 12 can be installed at various locations on the mobile object 30. In one embodiment, the first camera 11 and the second camera 12 are installed inside the mobile object 30, which is a vehicle, and can capture images of the exterior of the mobile object 30 through the windshield. For example, the first camera 11 and the second camera 12 are arranged in front of the rearview mirror or on the dashboard. In one embodiment, the first camera 11 and the second camera 12 can be fixed to any of the vehicle's front bumper, fenders, side fenders, light modules, and hood.

[0077] The object detection device 20 can be located anywhere within the mobile object 30. For example, the object detection device 20 can be located within the dashboard of the mobile object 30. The object detection device 20 acquires a first image and a second image from the stereo camera 10. The object detection device 20 detects an object based on the first and second images. If the mobile object 30 is a vehicle, the object that the object detection device 20 detects can be an object on the road surface. Examples of such objects on the road surface include other vehicles and pedestrians.

[0078] The object detection device 20 can be configured to read a program recorded on a non-transitory computer-readable medium to implement the processing performed by the control unit 24 described below. Non-transitory computer-readable media include, but are not limited to, magnetic storage media, optical storage media, optical magnetic storage media, and semiconductor storage media. Magnetic storage media include magnetic disks, hard disks, and magnetic tapes. Optical storage media include optical disks such as CDs (Compact Discs), DVDs, and Blu-ray Discs (Blu-ray (registered trademark) Discs). Semiconductor storage media include ROMs (Read Only Memory), EEPROMs (ElecTrically Erasable Programmable Read-Only Memory), and flash memories.

[0079] The object detection device 20 includes an acquisition unit 21 , an output unit 22 , a memory 23 , and a control unit 24 (processor).

[0080] The acquisition unit 21 is an input interface of the object detection device 20. The acquisition unit 21 can receive information input from the stereo camera 10 and other devices. The acquisition unit 21 can use physical connectors and wireless communication devices. Physical connectors include electrical connectors corresponding to the transmission of electrical signals, optical connectors corresponding to the transmission of optical signals, and electromagnetic connectors corresponding to the transmission of electromagnetic waves. Electrical connectors include connectors based on IEC 60603, connectors based on USB standards, connectors corresponding to RCA terminals, connectors corresponding to S terminals specified in EIAJ CP-1211A, connectors corresponding to D terminals specified in EIAJ RC-5237, connectors based on HDMI (registered trademark) standards, and connectors corresponding to coaxial cables including BNC. Optical connectors include various connectors based on IEC 61754. Wireless communicators include wireless communicators based on various standards including Bluetooth (registered trademark) and IEEE 802.11. The wireless communicator includes at least one antenna.

[0081] The acquisition unit 21 may be input with image data of images captured by the first camera 11 and the second camera 12. The acquisition unit 21 transmits the input image data to the control unit 24. The acquisition unit 21 may correspond to the transmission method of the imaging signal of the stereo camera 10. The acquisition unit 21 may be connected to the output interface of the stereo camera 10 via a network.

[0082] The output unit 22 is an output interface of the object detection device 20. The output unit 22 can output the processing results of the object detection device 20 to other devices within the mobile object 30 or other devices outside the mobile object 30. Other devices within the mobile object 30 may include driving assistance devices such as automatic cruise control and safety devices such as automatic braking systems. Other devices outside the mobile object 30 may include other vehicles and road testers. Other devices within the mobile object 30 or other devices outside the mobile object 30 can appropriately use the information received from the object detection device 20. Similar to the acquisition unit 21, the output unit 22 can include various interfaces corresponding to wired and wireless communication. For example, the output unit 22 can have a CAN interface. In this case, the output unit 22 communicates with other devices within the mobile object 30.

[0083] The memory 23 stores programs used for various processes and information used in operations. The memory 23 includes a volatile memory and a non-volatile memory. The memory 23 includes a memory independent of the processor and a built-in memory of the processor.

[0084] The control unit 24 includes one or more processors. The processors include general-purpose processors for reading specific programs and executing specific functions and special-purpose processors specifically used for specific processing. Special-purpose processors include application-specific integrated circuits (ASICs). Processors include programmable logic devices (PLDs). PLDs include FPGAs (Field-Programmable Gate Arrays). The control unit 24 may include any one of an SoC (System-on-a-Chip) and a SiP (System In a Package) in which one or more processors cooperate. The processing performed by the control unit 24 can also be referred to as processing performed by a processor.

[0085] The control unit 24 performs various processes on the first disparity map during information processing within the object detection device 20. The first disparity map is formed by associating disparity with two-dimensional coordinates. The two-dimensional coordinates of the first disparity map are composed of a horizontal direction corresponding to the horizontal direction of the image captured by the stereo camera 10 and a vertical direction intersecting the horizontal direction. The horizontal direction is a first direction. The vertical direction is a second direction. The horizontal direction and the vertical direction may be orthogonal to each other. The horizontal direction may correspond to the width direction of the road surface. The horizontal direction may be a direction parallel to the horizontal line when the image captured by the stereo camera 10 contains a horizontal line. The vertical direction may be a direction corresponding to the direction in which gravity is applied in real space.

[0086] During information processing within the object detection device 20, the first parallax map undergoes various operations. These operations include computational processing and writing to and reading from the memory 23. The image resulting from the first parallax map is also referred to as a "first parallax image." A first parallax image is an image in which pixels representing parallax are arranged on a two-dimensional plane defined by horizontal and vertical dimensions. The following describes how the control unit 24 performs various operations on the first parallax image. In the following description, processing of the first parallax image may also be referred to as processing of the first parallax map.

[0087] The structure of the object detection system 1 of the present disclosure is not limited to Figure 1 The structure shown. Figure 4An object detection system 1A according to another embodiment of the present disclosure is shown. The object detection system 1A includes a stereo camera 10, an object detection device 20, and a generation device 25. The generation device 25 can be installed on hardware separate from the object detection device 20. The generation device 25 generates a first parallax image based on the first image and the second image output from the stereo camera 10. The generation device 25 includes a processor. The processor included in the generation device 25 generates the first parallax image based on the first image and the second image output from the first camera 11 and the second camera 12 of the stereo camera 10, respectively. The acquisition unit 21 acquires the first parallax image from the generation device 25. The object detection device 20 and the generation device 25 can be collectively regarded as one object detection device.

[0088] Below, refer to Figure 5 The illustrated flowchart explains the processing executed by the control section 24 . Figure 5 It is shown by Figure 1 Flowchart showing an example of the overall flow of processing executed by the object detection device 20 shown.

[0089] First, in Figure 5 Before describing in detail the processing performed in each step of the flowchart, an overview and purpose of the processing in each step will be briefly described.

[0090] Step S101 is a step of acquiring or generating a first parallax image. Step S101 is equivalent to the first processing described later. Figure 1 In the structure shown, the control unit 24 generates a first parallax image. Figure 4 In the illustrated structure, the control unit 24 acquires the first parallax image generated by the generating device 25 through the acquiring unit 21 .

[0091] Step S102 is the step of estimating the shape of the road surface. The process performed in step S102 is also referred to as the "first process." By estimating the road surface shape, the parallax corresponding to the road surface relative to the longitudinal coordinates can be estimated in the first parallax image. The road surface shape can be used in the following process to remove unnecessary parallax and / or estimate the height position of the road surface in real space.

[0092] Step S103 is a step of generating a second parallax image by removing unnecessary parallax from the first parallax image. The processing performed by step S103 is also referred to as "second processing." Unnecessary parallax includes parallax corresponding to white lines on the road surface and parallax corresponding to structures above the road surface, which may be included in the first parallax image. By removing unnecessary parallax from the first parallax image, the possibility of the object detection device 20 mistakenly detecting white lines on the road surface and structures above the road surface as objects to be detected on the road surface is reduced. As a result, the accuracy of the object detection device 20 in detecting objects can be improved.

[0093] Step S103 may be a step of generating a second disparity map by removing unnecessary disparity from the first disparity map during information processing within object detection device 20. The second disparity image is obtained by converting the second disparity map into an image. In the following description, processing the second disparity image may also be referred to as processing the second disparity map.

[0094] Step S104 is a step of detecting the first disparity and the second disparity based on the second disparity image. The first disparity is detected as the disparity corresponding to the object to be detected. The second disparity is detected as a candidate for the disparity corresponding to the object to be detected. Whether to restore the second disparity as the disparity corresponding to the object to be detected can be determined in the restoration process described later.

[0095] Step S105 is a step of calculating the height of the object on the image. The height of the object on the image detected can be Figure 51 The height of the detection frame 182 shown. The height of the object on the image is also referred to as the “height of the detection frame.” The process executed in step S105 is also referred to as “height calculation processing.”

[0096] Step S106 is a step of detecting the parallax corresponding to an object parallel to the direction of travel of the mobile body 30 in the second parallax image. An object parallel to the direction of travel of the mobile body 30 is also referred to as a "parallel object". As an example of a parallel object, structures at the end of the road such as guardrails and soundproof walls on highways or the sides of other vehicles can be cited. For example, there are cases where parallel objects such as guardrails and objects to be detected such as other vehicles are close to each other in the second parallax image. Parallel objects are detected, and, for example, pre-marking the detected parallel objects can reduce the possibility that the object detection device 20 will mistakenly detect parallel objects as objects to be detected on the road surface. With such a structure, the accuracy of the object detection device 20 in detecting objects can be improved. The processing performed by step S106 is also referred to as "parallel object detection processing."

[0097] Step S107 is a step of determining whether the second parallax is restored to the parallax corresponding to the object to be detected.

[0098] Step S108 determines the representative parallax at each horizontal coordinate in the second parallax image based on the first parallax and the restored second parallax. The processing performed in step S108 is also referred to as the "third processing." The processing performed in steps S104 to S108 is also referred to as the "third processing."

[0099] Step S109 is a step for detecting objects by converting information representing parallax into coordinates in real space and extracting clusters (groups) representing parallax. The processing performed in step S109 is also referred to as the "fourth processing." In step S109, the position of the object to be detected and the width of the object as viewed from the stereo camera 10 are obtained.

[0100] Step S110 is a step of outputting information about the detected object from the output unit 22. The object information output in step S110 may include the height of the object on the image calculated in step S105, the position of the object detected in step S109, and the width of the object as seen from the stereo camera 10 side detected in step S109. This information may be provided to other devices within the mobile object 30.

[0101] Next, the details of each step will be described.

[0102] In the process of step S101, the control unit 24 acquires or generates a first parallax image. Figure 1 In the object detection system 1 shown in FIG. 1 , the control unit 24 generates a first parallax image based on the first image and the second image acquired by the acquisition unit 21 . Figure 4 In the object detection system 1A shown, the control unit 24 acquires the first parallax image generated by the generating device 25 via the acquiring unit 21. The control unit 24 may store the first parallax image in the memory 23 for subsequent processing.

[0103] Since the method for generating the first parallax image is well known, a brief description will be given below. In the following, it is assumed that the control unit 24 generates the first parallax image.

[0104] The control unit 24 obtains a first image captured by the first camera 11 and a second image captured by the second camera 12. The control unit 24 divides one of the first and second images (for example, the first image) into a plurality of small regions. A small region can be a rectangular region with multiple pixels arranged vertically and horizontally. For example, a small region can be composed of three pixels arranged vertically and three pixels arranged horizontally. The number of pixels arranged vertically and horizontally in a small region is not limited to three. In addition, the number of pixels in the vertical and horizontal directions of a small region can be different. The control unit 24 shifts the pixels of each of the divided small regions horizontally by one pixel at a time on the other image and compares their features to perform matching. For example, when the first image is divided into small regions, the control unit 24 shifts the small regions of the first image horizontally by one pixel at a time on the second image and compares their features to perform matching. Features are, for example, brightness and color patterns. A method using the SAD (Sum of Absolute Difference) function is known for matching stereo images. This represents the sum of the absolute values ​​of the differences in luminance values ​​within a small area. When the SAD function is minimum, the two images are judged to be most similar. Stereo image matching is not limited to methods using the SAD function. Other methods can also be used for stereo image matching.

[0105] The control unit 24 calculates the disparity of each small area based on the difference in the horizontal position of the pixels of the two areas after matching the first image and the second image. The disparity can be the difference between the position of the same subject in the first image and the second image. The size of the disparity can be expressed in units of the horizontal width of the pixels on the stereo image. The size of the disparity can be calculated with an accuracy of less than one pixel through interpolation processing. The size of the disparity corresponds to the distance between the subject photographed by the stereo camera 10 and the stereo camera 10 in the real space. The closer the distance from the stereo camera 10 in the real space to the subject, the larger the disparity corresponding to the subject. The farther the distance from the stereo camera 10 in the real space to the subject, the smaller the disparity corresponding to the subject.

[0106] The control unit 24 generates a first parallax image representing the distribution of the calculated parallax. Pixels representing the parallax constituting the first parallax image are also referred to as "parallax pixels." The control unit 24 can generate the first parallax image with the same resolution as the pixels of the original first and second images.

[0107] Figure 6. The first parallax image 40 is shown in the figure. The first parallax image 40 is a two-dimensional plane composed of the horizontal direction (first direction) of the stereo camera 10 and the vertical direction (second direction) orthogonal to the horizontal direction. Parallax pixels representing parallax are arranged in the two-dimensional plane of the first parallax image 40. The horizontal direction is also called the "u direction". The vertical direction is also called the "v direction". The coordinate system composed of the u direction and the v direction is also called the "uv coordinate system" and the "image coordinate system". In this embodiment, the corner facing the upper left side of the paper of each figure becomes the origin (0, 0) of the uv coordinate system. In addition, the direction from the left side to the right side of the paper of each figure is the positive direction of the u axis, and the opposite direction is the negative direction of the u axis. In addition, the direction from the top to the bottom side of the paper of each figure is the positive direction of the v axis, and the opposite direction is the negative direction of the v axis. The negative direction of the v axis corresponds to the direction upward from the road surface in real space. The u coordinate and the v coordinate can be expressed in units of parallax pixels.

[0108] like Figure 6 As shown, first parallax image 40 includes parallax image 41, parallax image 42, and parallax image 43. Parallax image 41 corresponds to the road surface in front of moving object 30. Parallax image 42 corresponds to another vehicle in front of moving object 30. Parallax image 43 is a parallax image corresponding to a guardrail.

[0109] The control unit 24 can display the parallax information included in each pixel of the first parallax image by the brightness or color of each pixel. Figure 6 In the first parallax image 40 shown, the parallax of each pixel is displayed by different shades for the sake of explanation. In the first parallax image 40, the darker the shade, the smaller the parallax represented by the pixels in the area. The lighter the shade, the larger the parallax represented by the pixels in the area. In the first parallax image 40, pixels in areas with the same shade each represent a parallax within a specified range. In an actual first parallax image, it is sometimes difficult to calculate the parallax in the above-mentioned matching process for calculating the parallax because the pixels in a certain area have fewer feature quantities on the stereo image than the pixels in other areas. It is difficult to calculate the parallax of a part of a spatially uniform subject, such as a vehicle window, and a part where white spots occur due to reflection of sunlight. In the first parallax image, when there is parallax corresponding to an object or structure, the object or structure can be displayed with a brightness or color different from the parallax of the background located farther away.

[0110] The control unit 24 may not display the first parallax image as an image after calculating the parallax. In other words, the control unit 24 only needs to retain the first parallax map that is the basis of the first parallax image and perform appropriate processing on the first parallax map.

[0111] After executing step S101, the control unit 24 performs a first process of estimating the shape of the road surface based on the first parallax image (step S102). Figure 7 、 Figure 8 and Figure 14 The process of estimating the road shape executed by the control unit 24 is described with reference to the flowchart of FIG. First, the control unit 24 extracts a road surface candidate parallax d from the first parallax image. c (Step S201). Road surface candidate disparity d c is the road disparity d collected from the first disparity image r The parallax of the road surface is likely to be high. r Means the parallax of the road area. Road parallax d r The parallax of objects on the road is not included. r Indicates the distance to the corresponding point on the road. Road parallax d r These are collected as disparities having similar values ​​at positions with the same v coordinate.

[0112] Road surface candidate disparity d c The details of the extraction process are as follows Figure 8 As shown in the flow chart. Figure 8 As shown, the control unit 24 calculates the initial value of the parallax used to calculate the road candidate parallax, that is, the road candidate parallax initial value d0, based on the installation position of the stereo camera 10 (step S301). The road candidate parallax initial value d0 is the initial value of the road candidate parallax at the position closest to the stereo camera 10 for extracting the road candidate parallax. The road candidate parallax extraction position closest to the stereo camera 10 can be set, for example, within a range of 1 to 10 meters from the stereo camera 10.

[0113] like Figure 9 As shown, in the stereo camera 10, the road surface height Y is the height of the stereo camera 10 in the vertical direction from the photographed road surface 41A. In addition, the road surface height Y0 is the height from the road surface 41A at the setting position of the stereo camera 10. Due to the undulations of the road, the road surface height Y sometimes varies depending on the distance from the stereo camera 10. Therefore, the road surface height Y at a position far away from the stereo camera 10 is inconsistent with the road surface height Y0 at the setting position of the stereo camera 10. In one embodiment of the multiple embodiments, it is assumed that the first camera 11 and the second camera 12 of the stereo camera 10 are set with the optical axis OX parallel to each other facing forward. Figure 9 In the equation, the distance Z represents the horizontal distance to a specific road surface position. Let the baseline length of the stereo camera 10 be B, and the vertical image size be TOTALv. In this case, the road parallax d of the road surface 41A captured at a certain vertical coordinate (v coordinate) is sThe relationship between the height of the road surface Y does not depend on the horizontal distance Z, but is given by the following formula.

[0114] d s =B / Y×(v-TOTALv / 2) (1)

[0115] The road parallax d calculated by equation (1) s It is also called "geometrically estimated road parallax". s Indicates the geometrically estimated road surface parallax.

[0116] The initial value d0 of the candidate road disparity is the road disparity between the stereo camera 10 and the candidate road disparity d closest to the stereo camera 10, assuming that the road 41A is located at a certain position. c The extraction positions are calculated when the optical axis OX of the first camera 11 and the second camera 12 is parallel and flat. In this case, the v coordinate on the first parallax image of the extraction position of the road candidate parallax closest to the position of the stereo camera 10 is determined as a specific coordinate (v0). The coordinate (v0) is the initial value of the v coordinate for extracting the road candidate parallax. The coordinate (v0) is between TOTALv / 2 and TOTALv. The coordinate (v0) is located at the bottom (the side with the larger v coordinate) within the range of image coordinates that can calculate the parallax. The coordinate (v0) can be used as TOTALv corresponding to the bottom row of the first parallax image. The initial value d0 of the road candidate parallax can be determined by substituting v0 into v in formula (1) and substituting Y0 into Y.

[0117] The control unit 24 calculates the disparity collection threshold for the first row whose vertical v coordinate is the coordinate (v0) based on the road surface candidate disparity initial value d0 (step S302). A row refers to the arrangement of pixels with the same v coordinate arranged horizontally on the first disparity image. The disparity collection threshold includes an upper threshold, which is the upper limit threshold for collecting disparities, and a lower threshold, which is the lower limit threshold for collecting disparities. The disparity collection threshold is set above and below the road surface candidate disparity initial value d0 based on a predetermined rule so as to include the road surface candidate disparity initial value d0. Specifically, the road surface parallax when the road surface height Y is varied by a predetermined road surface height change ΔY from the state where the road surface candidate disparity initial value d0 is calculated is determined as the upper and lower thresholds of the disparity collection threshold. In other words, the lower threshold of the disparity collection threshold is obtained by subtracting the disparity corresponding to the road surface height change ΔY from the road surface candidate disparity initial value d0. The upper threshold of the disparity collection threshold is obtained by adding the disparity corresponding to the road surface height change ΔY to the road surface candidate disparity initial value d0. Specifically, the lower limit threshold and the upper limit threshold of the parallax collection threshold are obtained by changing the value of Y in the equation (1).

[0118] Thereafter, the control unit 24 repeats the processes from step S303 to step S307. First, the control unit 24 processes the row whose v coordinate is the coordinate (v0) located at the bottom of the first parallax image (step S303).

[0119] The control unit 24 collects disparities using the disparity collection threshold (step S304). The control unit 24 collects disparity pixels having a disparity between the lower threshold and the upper threshold of the disparity collection threshold as road surface candidate disparities d c That is, the control unit 24 determines that the disparity pixels having the disparity falling within the predetermined limit range based on the road surface candidate disparity initial value d0 calculated using the equation (1) are candidates for the disparity pixels representing the correct disparity of the road surface 41A. The control unit 24 sets the disparity of the disparity pixels determined as candidates for the disparity pixels representing the correct disparity of the road surface 41A as the road surface candidate disparity d c This configuration reduces the possibility that the control unit 24 will mistakenly determine the parallax corresponding to objects or structures other than the road surface 41A as the parallax corresponding to the road surface 41A. This improves the accuracy of detecting the road surface 41A.

[0120] In the process of step S304, when the determination of all the parallax pixels whose v coordinate is the coordinate (v0) is completed, the control unit 24 calculates the collected road surface candidate parallax d c Average and calculate the road candidate disparity d c The average value of the candidate road disparity d av (Step S305). The control unit 24 can c And its uv coordinates, and v coordinates are the average road candidate disparity d under the coordinate (v0) av Stored in memory 23.

[0121] After executing the process of step S305, the control unit 24 executes the process of step S306. In the process of step S306, the control unit 24 calculates the average road surface candidate parallax d of the v coordinate (v0) calculated in the process of step S305. av The control unit 24 calculates the parallax collection threshold for each parallax pixel of the row above, that is, the row with the v coordinate of the coordinate (v0-1). av The road surface height Y is changed so that the equation (1) is satisfied. The control unit 24 substitutes v0-1 for v0 in the equation (1) with the changed road surface height Y, and calculates the geometrically estimated road surface parallax d when the v coordinate is the coordinate (v0-1). sSimilar to the process of step S302, the control unit 24 can use the geometrically estimated road surface parallax d s The control unit 24 can use the geometrically estimated road surface parallax d as the lower limit threshold of the parallax collection threshold after subtracting the parallax corresponding to the predetermined road surface height change ΔY. s The parallax obtained by adding the parallax corresponding to the predetermined road surface height change ΔY is used as the upper limit threshold of the parallax collection threshold.

[0122] After executing the process of step S306, the control unit 24 determines the geometrically estimated road surface parallax d calculated by equation (1). s Is it greater than a specified value. The specified value is, for example, one pixel. s If it is greater than 1, the control unit 24 returns to the process of step S303 (step S307). In the process of step S303, the control unit 24 sets the road surface candidate disparity d c The extracted object moves to a row of one pixel upward. That is, when the road candidate disparity d c When the extraction target is a row with v coordinate (v0), the control unit 24 changes the v coordinate of the row of the target detected by the road surface to the coordinate (v0-1). Figure 10 As shown, when the road candidate disparity d c When the calculation target of is the n-th row, the control unit 24 changes the target row of road surface detection to the n+1-th row. Figure 10 For illustration purposes, the vertical width of each row is exaggerated. Each row is actually one pixel high. In this case, the v coordinate of the n+1th row is 1 less than the v coordinate of the nth row.

[0123] The processing of steps S304 to S306 for the n+1th row is performed in the same or similar manner as the processing of the row whose v coordinate is the coordinate (v0). In the processing of step S304, the control unit 24 collects the road surface candidate disparity d using the disparity collection threshold value calculated in the processing of step S306 for the nth row. c In the process of step S305, the control unit 24 processes the collected road surface candidate disparity d c Average and calculate the average road candidate disparity d av In the process of step S306, the control unit 24 uses the average road surface candidate parallax d av The control unit 24 calculates the geometrically estimated road surface parallax d using the formula (1) after the road surface height Y is changed. s Furthermore, the control unit 24 extracts the road surface candidate disparity d of the n+2th row. c , in the geometric estimation of road parallax d s The parallax collection threshold is calculated by taking into account the road height change ΔY.

[0124] The control unit 24 sets the road surface candidate disparity d c The object to be extracted is the candidate disparity d of the road surface closest to the stereo camera 10 c While shifting upward (the negative direction of the v coordinate) from the row corresponding to the extraction position, the candidate road disparity d corresponding to the v coordinate is extracted. c The control unit 24 can extract the road candidate disparity d c The average road candidate disparity d corresponding to the corresponding u coordinate and v coordinate and the v coordinate av Stored together in the memory 23.

[0125] The control unit 24 calculates the geometrically estimated road surface parallax d calculated by equation (1) in step S307. s When the road surface candidate parallax d is less than the above-mentioned predetermined value, the c The extraction process and return to Figure 7 The predetermined value can be set to one pixel, for example.

[0126] In this way, Figure 8 In the flowchart, the road candidate disparity d is extracted c The initial value of the v coordinate is set to v0 corresponding to the position of the near distance side viewed from the stereo camera 10, and the candidate disparity d of the road surface on the far distance side is extracted in sequence. c Generally, the detection accuracy of the parallax on the short distance side of the stereo camera 10 is higher than that of the parallax on the long distance side. Therefore, by sequentially extracting the road surface candidate parallax d from the short distance side to the long distance side, c , which can improve the detected road candidate disparity d c precision.

[0127] When extracting the candidate disparity d c of Figure 8 In the flowchart, the road candidate disparity d is calculated for each longitudinal coordinate. c In other words, when extracting the above-mentioned road candidate disparity d c In the flowchart, the road candidate disparity d is calculated for each pixel row in the vertical direction. c The unit for calculating the road candidate disparity is not limited to this. It is also possible to calculate the road candidate disparity d by combining multiple coordinates in the vertical direction. c .

[0128] In steps S301 to S307, the candidate road surface disparity d c After the extraction process, the control unit 24 enters Figure 7 The control unit 24 estimates the road surface parallax d in order from the short distance side to the long distance side. rWhen the road parallax d r To this end, first, the control unit 24 initializes the Kalman filter (step S202). As the initial value of the Kalman filter, the road parallax d calculated in the process of step S305 can be used. r The average road surface candidate disparity d corresponding to the bottom row (row with v coordinate value v0) among the estimated rows av The value of .

[0129] The control unit 24 sequentially executes the following processes of steps S203 to S210 while changing the target row from the short distance side to the long distance side of the road surface (step S203 ).

[0130] First, the control unit 24 selects a target line in the first parallax image based on the road surface candidate parallax d within a range of a fixed width in the real space. c Generate a representation of each road disparity d r The frequency histogram of the values ​​of (step S204). The fixed width range in the actual space refers to the range of the width of the driving lane of the road. The fixed width can be set to a value such as 2.5m or 3.5m. The range of obtaining the parallax is initially set to be, for example, the range of the parallax obtained by Figure 11 The range surrounded by the solid frame line 45. The fixed width is pre-stored in the memory 23 of the object detection device 20. By limiting the range of obtaining parallax to this range, the possibility of the control unit 24 mistaking objects other than the road surface 41A or structures such as soundproof walls for the road surface 41A and extracting them is reduced. This can improve the accuracy of road surface detection. Figure 11 The range for obtaining the parallax indicated by the solid line can be sequentially changed from the initially set frame line 45 according to the situation on the road ahead as described later.

[0131] The control unit 24 uses the Kalman filter to calculate the road parallax d r The road parallax d is set for the target row. r The acquisition range of road parallax d r The acquisition range is based on the Kalman filter to predict the road parallax d of the next line r The reliability is determined by the variance σ of the Gaussian distribution. 2 (σ is the road parallax d r The control unit 24 can calculate the road parallax d using the predicted value ±2σ or the like. r The control unit 24 generates the road surface candidate parallax d generated in step S204. c The histogram of the road parallax d is set based on the Kalman filter. rThe road parallax d with the largest frequency is extracted within the acquisition range r The control unit 24 extracts the road parallax d r Let the road parallax d of the target row be r Observation value (step S205).

[0132] Next, the control unit 24 checks the road surface parallax d determined in the process of step S205. r The correct road parallax d does not include the parallax corresponding to the object. r (Step S206) The control unit 24 calculates all the road surface parallaxes d detected in each row up to the row currently being processed. r , generate a map to convert the road parallax d r The dv correlation diagram is obtained on the dv coordinate space with the v coordinate as the coordinate axis. When the road surface 41A is correctly detected, the dv correlation diagram is Figure 12 As shown by the dotted line, as the value of v coordinate decreases, the road parallax d r It also decreases linearly.

[0133] On the other hand, if the parallax representing the object is mistakenly recognized as the parallax representing the road surface 41A, as shown in FIG. Figure 13 As shown in the dv correlation diagram, in the part representing the parallax of the object, the parallax d is approximately constant and has nothing to do with the change of the longitudinal coordinate (v coordinate). Generally, the object includes a part perpendicular to the road surface 41A, so it is displayed as containing a large number of equidistant parallaxes on the first parallax image. Figure 13 In the example, the parallax d of the first portion R1 decreases as the v coordinate value changes. The first portion R1 correctly detects the parallax representing road surface 41A. The parallax d of the second portion R2 remains constant even when the v coordinate changes. The second portion R2 is considered to be the portion where the parallax representing an object is mistakenly detected. The control unit 24 can determine that the parallax representing an object was mistakenly recognized as the parallax representing road surface 41A when a predetermined number of lines with approximately equal parallax d values ​​persist.

[0134] When the parallax is determined not to be the correct road parallax d in the process of step S206 r When (step S206: No), the control unit 24 searches again for the road surface parallax d starting from the line where it is determined that the parallax representing the object is erroneously detected. r (Step S207). In the process of step S207, the control unit 24 re-searches the road parallax histogram in the area of ​​the row where the parallax d does not change even if the value of the v coordinate changes. In this area, if there is a high frequency of parallax in the part with a smaller parallax than the parallax d determined in the process of step S205, the control unit 24 can determine that the parallax is the correct road parallax d. r Observed values.

[0135] In the process of step S206, the road parallax d r If it is determined to be correct (step S206: Yes) or in the process of step S207, the road surface parallax d r When the re-search is completed, the control unit 24 proceeds to the process of step S208. In the process of step S208, the control unit 24 determines the lateral range of the road surface 41 on the first parallax image, which is the target of generating the next row of histograms shifted by one pixel in the vertical direction. For example, Figure 11 As shown, when the parallax image 42 corresponding to another vehicle exists on the parallax image 41 corresponding to the road surface, the control unit 24 cannot obtain the correct road surface parallax d of the parallax image 41 corresponding to the road surface that overlaps with the other vehicle. r If the road parallax d can be obtained r The range of the parallax image 41 corresponding to the road surface becomes narrower, and it is difficult for the control unit 24 to obtain the correct road surface parallax d r Therefore, if Figure 11 As shown by the middle dotted line, the control unit 24 acquires the road surface candidate parallax d c Specifically, when the control unit 24 determines that the parallax representing the object is included in the process of step S206, it detects which side of the object represents the correct road parallax d. r The candidate disparity of the road surface d c In the next row, the inclusion of the horizontal direction indicates more correct road parallax d r The candidate disparity of the road surface d c On one side (on Figure 11 The range of parallax is obtained by moving the image (center right) in sequence.

[0136] Next, the control unit 24 uses the road surface parallax d of the current line determined by the process of step S205 or the process of S207. r To update the Kalman filter (step S209). That is, the Kalman filter is based on the road parallax d of the current row. r The road parallax d is calculated from the observation value r When calculating the estimated value of the current row, the control unit 24 calculates the road parallax d of the current row. r The estimated value of is added as part of the past data and used to calculate the road parallax d of the next row. r Considering that the height of the road surface 41A does not suddenly change up and down relative to the horizontal distance Z from the stereo camera 10, the estimation using the Kalman filter in this embodiment is estimated to be the road surface parallax d in the current row. r There is a road parallax d of the next row near rIn this way, the control unit 24 sets the road parallax d of the current line r By limiting the parallax range for generating the next row of histograms to the vicinity of , the possibility of erroneous detection of objects other than the road surface 41A is reduced. In addition, the amount of calculations executed by the control unit 24 can be reduced and the processing speed can be increased.

[0137] The road surface parallax d estimated by the Kalman filter in step S209 r If the parallax d estimated by the Kalman filter is greater than the predetermined value, the control unit 24 returns to the process of step S203 and repeats and executes the processes of steps S203 to S209. r If the value is less than or equal to the predetermined value (step S210), the control unit 24 proceeds to the next process (step S211). The predetermined value can be set to, for example, one pixel.

[0138] In the process of step S211, the control unit 24 uses two straight lines on the dv correlation diagram to correlate the longitudinal image coordinate v and the estimated road surface parallax d. r The relationship between the road parallax d is approximated. r The value of the v coordinate is related to the distance Z from the stereo camera 10 and the road surface height Y. Therefore, the v coordinate and the road surface parallax d are connected by two straight lines. r The relationship between the distance Z from the stereo camera 10 and the height of the road surface 41A can be approximated by two straight lines. Figure 14 The flowchart of FIG. 1 illustrates the processing of step S211 in detail.

[0139] First, use Figure 7 The process up to step S210 obtains the road parallax d r and the correlation between the v coordinate. For example, the v coordinate and the road parallax d r The correlation between Figure 15 The dashed line graph 51 is shown in the dv coordinate space. In real space, if road surface 41A is flat and has no inclination changes, graph 51 is a straight line. However, the inclination of road surface 41A in reality may change due to ups and downs, such as inclines. If the inclination of road surface 41A changes, graph 51 in the dv coordinate space cannot be represented by a straight line. Attempting to approximate the inclination of road surface 41A using three or more straight lines or curves increases the processing load on object detection device 20. Therefore, in this application, graph 51 is approximated using two straight lines.

[0140] like Figure 15As shown, the control unit 24 calculates the estimated road surface parallax d on the lower side (near distance side) in the dv coordinate space by the least square method using the first straight line 52. r Approximation is performed (step S401). The approximation performed by the first straight line 52 can be performed within the distance range in which the object detection device 20 is to perform object detection, and the road parallax d corresponding to the predetermined distance. r The predetermined distance may be set to half the distance range within which the object detection device 20 is to detect objects. For example, if the object detection device 20 is designed to detect objects at a maximum distance of 100 meters, the first straight line 52 may be determined using the least squares method to best approximate the pattern 51 within the range from the closest distance measurable by the stereo camera 10 to a distance of 50 meters.

[0141] Next, the control unit 24 determines whether the inclination of the road surface 41A represented by the first straight line 52 approximated in the processing of step S401 is a possible inclination of the road surface 41A (step S402). The inclination angle of the first straight line 52 is a plane when converted into real space. The inclination of the first straight line 52 corresponds to the inclination angle of the road surface 41A in the yz plane determined based on conditions such as the road surface height Y0 and the baseline length B at the installation position of the stereo camera 10. The control unit 24 can determine that it is a possible inclination when the inclination of the road surface 41A in real space corresponding to the first straight line 52 is within a range of predetermined angles based on the horizontal plane in real space. The control unit 24 can determine that it is an impossible inclination when the inclination of the road surface 41A in real space corresponding to the first straight line 52 is outside the range of predetermined angles based on the horizontal plane in real space. The predetermined angle can be appropriately set in consideration of the driving environment of the mobile object 30.

[0142] In the process of step S402, if it is determined that the inclination of the first straight line 52 is an inclination that is impossible for the road surface 41A to exist (step S402: No), the control unit 24 determines the first straight line 52 based on a theoretical road surface assuming that the road surface 41A is flat (step S403). The theoretical road surface can be calculated based on the installation conditions such as the road surface height Y0, the installation angle, and the baseline length B at the installation position of the stereo camera 10. When the road surface parallax d calculated from the image is r If the parallax is unreliable, the control unit 24 uses the road parallax of the theoretical road. For example, the control unit 24 mistakenly extracts the parallax of an object or structure other than the road surface 41A as the road parallax d r In the case of , it is possible to determine that the road surface 41A has an unrealistic inclination and eliminate the error. This can reduce the possibility of misjudging the parallax of objects or structures other than the road surface 41A as the road surface parallax d. r possibility.

[0143] If, in step S402, the control unit 24 determines that the inclination of the first straight line 52 is an inclination that is impossible for the road surface 41A (step S402: Yes), or after executing step S403, the control unit 24 proceeds to step S404. In step S404, the control unit 24 determines an approximation starting point 53 for starting approximation of the second straight line 55. The control unit 24 calculates the approximation error with respect to the graph 51 from the minimum side (far side) to the maximum side (near side) of the v coordinate of the first straight line 52 and can select the coordinate on the first straight line 52 where the approximation error is continuously less than a predetermined value as the approximation starting point 53. Alternatively, the approximation starting point 53 can be determined as the coordinate on the first straight line 52 where the approximation error is greater than the predetermined value by calculating the approximation error with respect to the graph 51 from the maximum side (near side) to the minimum side (far side) of the v coordinate of the first straight line 52. The v coordinate of the approximation starting point 53 is not fixed to a specific value. Approximation starting point 53 can be set on first straight line 52 at a position corresponding to the v-coordinate of a position closer to stereo camera 10 than half the distance range within which object detection device 20 performs object detection. For example, if first straight line 52 approximates road surface 41A from the closest measurable distance to a distance of 50 meters, approximation starting point 53 can be set at a position corresponding to the v-coordinate 40 meters before 50 meters.

[0144] After executing the process of step S404, the control unit 24 repeatedly executes the processes of steps S405 to S407. Figure 16 As shown, the control unit 24 sets the angle difference from the first straight line 52 to an angle selected from a predetermined angle range and sequentially selects candidate straight lines 54, which are candidates for the second straight line 55 starting from the approximation starting point 53 (step S405). The predetermined angle range is set to an angle that allows the road slope to vary within the distance range of the measurement target. The predetermined angle range can be, for example, ±3 degrees. For example, the control unit 24 may adjust the angle of the candidate straight line 54 to an angle of -3 degrees from the angle of the first straight line 52 to +3 degrees, in increments of 0.001 degrees.

[0145] For each selected candidate line 54, the control unit 24 calculates the error between the portion located above (farther away from) the approximate starting point 53 of the graphic 51 in the dv coordinate space (step S406). The error can be calculated using the mean square error of the parallax d relative to the v coordinate. The control unit 24 stores the calculated error for each candidate line 54 in the memory 23.

[0146] When the control unit 24 finishes calculating the errors for all candidate straight lines 54 within the angle range (step S407), it searches for the minimum error among the errors stored in the memory 23. Figure 17 As shown, the control unit 24 selects the candidate straight line 54 having the smallest error as the second straight line 55 (step S408).

[0147] When the second straight line 55 is determined in the process of step S408, the control unit 24 determines whether the error between the second straight line 55 and the graph 51 is within a predetermined value (step S409). The predetermined value is appropriately set to obtain desired road surface estimation accuracy.

[0148] In the process of step S409, if the error is within the prescribed value (step S409: Yes), the road surface parallax d is approximated using the first straight line 52 and the second straight line 55. r .

[0149] In the process of step S409, if the error exceeds the predetermined value (step S409: No), the control unit 24 extends the first straight line 52 upward (far distance side) and rewrites the approximation result (step S410). As described above, the road surface parallax d is approximated by two straight lines. r .

[0150] Approximate the road parallax d relative to the v coordinate using two straight lines r , thereby approximating the road shape using two straight lines. This reduces the load associated with subsequent calculations and speeds up object detection compared to approximating the road shape using a curve or three or more straight lines. Furthermore, compared to approximating the road shape using a single straight line, the error with the actual road surface is smaller. Furthermore, by not fixing the v coordinate of the approximation starting point 53 of the second straight line 55 to a predetermined coordinate, the accuracy of the approximation with the actual road surface can be improved compared to a case where the coordinates of the approximation starting point 53 are fixed in advance.

[0151] In the process of step S409, if the error is within the prescribed value (step S409: Yes) or after the process of step S410 is executed, the control unit 24 calculates the road surface parallax d r The linear approximation process is completed and returns to Figure 7 The processing of step S212.

[0152] In the process of step S212, the road surface parallax d removed from the first parallax image is determined. r The road parallax d removed from the first parallax image is r The threshold value is equivalent to the first height described later. The first height can be used to remove the road parallax d in the processing of the next step S103. rCalculated in this way.

[0153] Then, the control unit 24 returns to Figure 5 By using the above process, the control unit 24 obtains the v coordinate in the dv coordinate space and the road parallax d r The approximate formula for the relationship between the v coordinate in the dv coordinate space and the road parallax d r The relationship between the distance Z in front of the stereo camera 10 and the road height Y in the real space is obtained by the approximate formula of the relationship. The control unit 24 performs the second processing (step S103) based on the approximate formula. The second processing is to remove the parallax corresponding to the range below the first height from the road surface 41A in the real space and the parallax corresponding to the subject above the second height from the road surface 41A from the first parallax image. Thus, the control unit 24 performs the second processing based on the approximate formula. Figure 6 The first parallax image 40 shown is generated Figure 18 The second parallax image 60 is shown. Figure 18 The figure is made for explanation. The actual second parallax image based on the image obtained from the stereo camera 10 is as follows Figure 19 As shown. Figure 19 In the image, the depth of black and white is used to express the size of the parallax. Figure 19 The second parallax image shown includes parallax images 44 corresponding to other vehicles.

[0154] The first height may be set to a value smaller than the minimum height of the object to be detected by the object detection device 20. The minimum height of the object to be detected by the object detection device 20 may be the height of a child (e.g., 50 cm). The first height may also be a value greater than 15 cm and less than 50 cm. Figure 6 The first parallax image 40 shown contains noise, and the aforementioned processing may reduce the accuracy of detecting the parallax corresponding to the road surface. In this case, simply removing the detected parallax corresponding to the road surface from the first parallax image 40 may leave the parallax corresponding to the road surface in a portion of the parallax image 41. By removing the parallax corresponding to the range below the first height from the road surface 41A from the first parallax image 40, a second parallax image is obtained, in which the parallax corresponding to the road surface is removed from the parallax image 41 with high accuracy.

[0155] exist Figure 18In the second parallax image 60 shown, parallax information corresponding to the range below the first height from the road surface in real space is removed. This structure eliminates parallax information from the parallax image 41 corresponding to the road surface. The parallax image 41 corresponding to the road surface is adjacent to the parallax image 42 corresponding to other vehicles. Since the second parallax image 60 does not include parallax information in the parallax image 41, subsequent processing of the parallax information in the parallax image 42 corresponding to other vehicles is facilitated. Furthermore, by eliminating unnecessary parallax unrelated to the object being detected, the processing speed described later can be increased.

[0156] The second height can be appropriately set based on the maximum value of the height of the object to be detected by the object detection device 20. In the case where the mobile body 30 is a vehicle, the second height can be set based on the upper limit of the height of the vehicle that can be driven on the road. The height of the vehicle that can be driven on the road is stipulated by traffic laws and regulations. For example, in Japan's Road Traffic Law, the height of a truck is, in principle, below 3.8m. In this case, the second height can be 4m. By removing the parallax corresponding to the object whose height from the road surface is above the second height from the first parallax image, the information of the parallax corresponding to the object is removed from the first parallax image. Figure 18 In the second parallax image 60 shown, the parallax corresponding to objects at a height higher than the second height from the road surface is removed, so the parallax image corresponding to objects on the negative side of the v-axis does not contain parallax information. Since the parallax information corresponding to objects at a height higher than the second height is removed, in subsequent processing, Figure 18 The processing of the parallax information of the parallax image 42 corresponding to other vehicles can be made easier. Furthermore, by removing unnecessary parallax that is not related to the object to be detected, the processing speed described later can be increased.

[0157] After executing the process of step S103, the control unit 24 detects the first parallax and the second parallax from the second parallax image (step S104). Figure 20 As shown in the flowchart shown.

[0158] In the process of step S501 , the control unit 24 divides the second parallax image into a plurality of partial regions by dividing it by Δu1 along the u direction. Figure 21In the figure, the partial area 61 is overlapped with the second parallax image 60. The partial area 61 can be a rectangle whose long side is significantly longer than the short side. The long side of the partial area 61 is more than 5 times the short side of the partial area 61. The long side of the partial area 61 can also be more than 10 times the short side of the partial area 61, or more than 20 times the short side of the partial area 61. The short side of the partial area 61, i.e., Δu1, can be several pixels to tens of pixels. The first parallax and the second parallax can be detected as described later based on the partial area 61. The shorter the short side of the partial area 61, i.e., Δu1, the higher the resolution of the detection of the first parallax and the second parallax can be. The control unit 24 can be from Figure 21 The partial regions 61 are sequentially acquired from the negative direction side to the positive direction side of the u-axis shown, and the following processes of step S502 to step S506 are executed.

[0159] In the process of step S502 , the control unit 24 generates a parallax histogram for each partial region. Figure 22 An example of a disparity histogram is shown in FIG. Figure 22 The horizontal axis corresponds to the magnitude of the parallax. Figure 22 The vertical axis corresponds to the frequency of parallax. The frequency of parallax is the number of parallax pixels representing the parallax contained in a partial area. Figure 21 As shown, the partial area 61 may include multiple parallax pixels in the parallax image corresponding to the same object. For example, the partial area 61-1 includes multiple parallax pixels 42a. The multiple parallax pixels 42a are the parallax pixels included in the partial area 61-1 among the multiple parallax pixels in the parallax image 42 corresponding to other vehicles. In the partial area 61, the parallax represented by the multiple parallax pixels corresponding to the same object may be roughly the same. For example, in the partial area 61-1, the parallax represented by the multiple parallax pixels 42a may be roughly the same. That is, in Figure 22 In the disparity histogram shown, the frequency of disparities corresponding to the same object may be high.

[0160] In the process of step S502, the control unit 24 may increase the value of the parallax as the parallax decreases. Figure 22The width of the interval Sn of the disparity histogram shown. Interval Sn is the interval of the nth disparity histogram counted from the side with the smaller parallax. The starting point of interval Sn is the parallax dn-1. The end point of interval Sn is the parallax dn. For example, the control unit 24 can make the width of interval Sn-1 larger than the width of interval Sn by about 10% of the width of interval Sn. Compared with the case where the distance from the stereo camera 10 to the object is short, when the distance from the stereo camera 10 to the object is long, the parallax corresponding to the object can be smaller. Compared with the case where the distance from the stereo camera 10 to the object is short, when the distance from the stereo camera 10 to the object is long, the number of pixels occupied by the object on the stereo image can be reduced. That is, compared with the case where the distance from the stereo camera 10 to the object is short, when the distance from the stereo camera 10 to the object is long, the number of parallax pixels representing the parallax corresponding to the object on the second parallax image can be reduced. As the parallax becomes smaller Figure 22 The width of the disparity histogram bin Sn is increased, thereby making it easier to detect the disparity corresponding to an object far from the stereo camera 10 by the processing of steps S503 to S506 described later. The control unit 24 may sequentially execute the processing of steps S503 to S506 for each bin of the disparity histogram.

[0161] In step S503, the control unit 24 determines whether there is a segment Sn in the generated disparity histogram where the frequency of disparity exceeds the first threshold Tr1 (a predetermined threshold). If the control unit 24 determines that there is a segment Sn in which the frequency of disparity exceeds the first threshold Tr1 (step S503: Yes), the control unit 24 detects the disparity in the range from the starting point of segment Sn (disparity dn-1) to the end point of segment Sn (disparity dn) as the first disparity (step S504). The control unit 24 detects the first disparity as the disparity corresponding to the object. The control unit 24 associates the detected first disparity with the u coordinate and stores it in the memory 23. On the other hand, if the control unit 24 determines that there is no segment Sn in the generated disparity histogram where the frequency of disparity exceeds the first threshold Tr1 (step S503: No), the process proceeds to step S505.

[0162] The first threshold Tr1 can be set based on the minimum value of the height of the object to be detected by the object detection device 20 (for example, the height of a child is 50 cm). However, as described above, when the object is far from the stereo camera 10, the parallax corresponding to the object can be smaller than when the object is close to the stereo camera 10. In this case, when the first threshold Tr1 is fixed with respect to the parallax, there is a case where it is difficult to detect the parallax corresponding to the object far from the stereo camera 10 compared to the object close to the stereo camera 10. Therefore, the first threshold Tr1 can be set based on the minimum value of the height of the object to be detected by the object detection device 20 and in a manner that increases as the parallax of the disparity histogram decreases. For example, the first threshold Tr1 can be calculated by formula (2).

[0163] Tr1=(D×H) / B (2)

[0164] In equation (2), parallax D is the parallax corresponding to the horizontal axis of the parallax histogram. Height H is the minimum height of the object to be detected by object detection device 20. Baseline length B is the distance (baseline length) between the optical centers of first camera 11 and second camera 12.

[0165] The process of step S505 is to determine whether to detect the second parallax. Before explaining the process of step S505, the reason for detecting the second parallax is explained. As mentioned above, in the matching process of calculating the parallax, due to the small amount of features on the stereo image, the parallax of a part of the parallax image corresponding to the object is not calculated or is smaller than the parallax of other parts. For example, Figure 19 As shown, since the feature quantity on the stereoscopic image is small, the parallax of the lower central part of the parallax image 44 corresponding to other vehicles is not calculated. When the parallax of a part of the parallax image corresponding to the object is not calculated, even if it is an object with an actual height of more than 50 cm in the actual space, it may be displayed as two or more separate objects with a height of less than 50 cm (for example, 10 cm) in the actual space on the second parallax image. That is to say, even if it is an object with an actual height in the actual space that is higher than the height of the detection object, there is a situation where the parallax corresponding to the object will not be detected as the first parallax through the determination process performed by the above-mentioned first threshold Tr1 when a part of the parallax image corresponding to the object is missing. Therefore, in the present embodiment, such a parallax that is not detected as the first parallax is preliminarily detected as a candidate for the parallax corresponding to the object, that is, as the second parallax. By preliminarily detecting it as the second parallax, in the later described Figure 5 In the process of step S107 described later, it can be determined whether the parallax of the object to be detected is restored.

[0166] In the process of step S505 , the control unit 24 determines whether there is a section Sn in which the frequency of parallax is less than or equal to the first threshold Tr1 and exceeds the second threshold Tr2 (predetermined threshold) in the generated parallax histogram.

[0167] The second threshold Tr2 may be a predetermined ratio of the first threshold Tr1. The predetermined ratio may be appropriately set based on the ratio between the portion where the parallax is not calculated and the portion where the parallax is calculated in the parallax image of the same object. For example, the predetermined ratio may be 0.2.

[0168] In the process of step S505, if the control unit 24 determines that there is a section Sn in which the frequency of parallax is less than the first threshold value Tr1 and exceeds the second threshold value Tr2 in the generated parallax histogram (step S505: Yes), the process proceeds to step S506. On the other hand, if the control unit 24 determines that there is no section Sn in which the frequency of parallax is less than the first threshold value Tr1 and exceeds the second threshold value Tr2 in the generated parallax histogram (step S505: No), the process returns to step S506. Figure 5 The process of step S105 is shown.

[0169] In the process of step S506, the control unit 24 detects the parallax in the range from the starting point of the interval Sn (parallax dn-1) to the end point of the interval Sn (parallax dn) as the second parallax (step S506). The control unit 24 associates the detected second parallax with the u coordinate and stores it in the memory 23. After executing the process of step S506, the control unit 24 returns to Figure 5 The process of step S105 is shown.

[0170] In the process of step S105, the control unit 24 calculates the height of the object on the image. The details of the process of step S105 are as follows: Figures 23 to 25 As shown in the flowchart shown.

[0171] Next, the control unit 24 performs object height calculation processing using the second disparity map, i.e., the second disparity image, as the disparity map. However, the control unit 24 can perform object height calculation processing on any disparity map in which disparity is associated with uv coordinates. In the object height calculation processing, the control unit 24 calculates the height of the object on the image by scanning disparity pixels along the v direction on the u coordinate of the search object. For example, the control unit 24 calculates the height of the object on the image by scanning disparity pixels representing the first disparity considered to correspond to the object and / or disparity pixels representing the second disparity that is a candidate for the disparity corresponding to the object along the v direction.

[0172] When the process of step S601 begins, the control unit 24 obtains the minimum u coordinates as the u coordinates of the search target. In this embodiment, the minimum and maximum coordinates, when used for scanning on an image, refer to the minimum and maximum coordinates of the scanning range. The minimum and maximum coordinates of the scanning range do not need to be consistent with the minimum and maximum coordinates on the image. The minimum and maximum coordinates of the scanning range can be set arbitrarily.

[0173] In the process of step S601, the control unit 24 refers to the memory 23 to determine whether there is a u-coordinate associated with the search object. Figure 20 The processing shown detects the first disparity or the second disparity.

[0174] Figure 26 The second parallax image 70 is shown in FIG. Figure 26 A portion of the second parallax image 70 is shown in FIG. The second parallax image 70 corresponds to Figure 18 Region 62 of second parallax image 60 is shown. Second parallax image 70 includes parallax image 42 of the other vehicle. In second parallax image 70, the shaded portion is a parallax pixel representing the first parallax or the second parallax. When the u-coordinate of the search object is coordinate (u0), control unit 24 determines that the first parallax and the second parallax associated with the u-coordinate of the search object do not exist. When the u-coordinate of the search object is coordinate (u1), control unit 24 determines that the first parallax or the second parallax associated with the u-coordinate of the search object exists.

[0175] In the process of step S601, if the control unit 24 does not determine that the first parallax or the second parallax associated with the u-coordinate of the search object exists (step S601: No), the process proceeds to step S602. On the other hand, if the control unit 24 determines that the first parallax or the second parallax associated with the u-coordinate of the search object exists (step S601: Yes), the process proceeds to step S604.

[0176] In the process of step S602, the control unit 24 determines whether the u coordinate of the search object is the maximum coordinate. When the u coordinate of the search object is the maximum coordinate, the control unit 24 ends the search for the u coordinate from the minimum coordinate to the maximum coordinate. When the control unit 24 determines that the u coordinate of the search object is the maximum coordinate (step S602: Yes), the control unit 24 ends the search. Figure 23 Process shown and return to Figure 5 For example, when the u coordinate of the search object is Figure 26When the coordinate (u0) shown in FIG6 is obtained, the control unit 24 determines that the u coordinate of the search object is not the maximum coordinate. On the other hand, when the control unit 24 determines that the u coordinate of the search object is not the maximum coordinate (step S602: No), the control unit 24 increases the u coordinate of the search object by 1 (step S603). The control unit 24 performs the process of step S601 on the u coordinate obtained by increasing the u coordinate by 1 in the process of step S603. For example, the control unit 24 can select the u coordinate of the search object from Figure 26 Starting from the coordinate (u0) shown in FIG. 1 , the processes of steps S601 to S603 are repeatedly performed until it is determined that the first parallax or the second parallax associated with the u coordinate of the search object exists.

[0177] In the process of step S604, the control unit 24 obtains the object parallax. The object parallax may be only the first parallax detected as the parallax corresponding to the object. In other words, the object parallax may be the parallax that satisfies the prescribed conditions. Figure 20 Alternatively, the object parallax may include both the first parallax and the second parallax. In other words, it may be a parallax that satisfies the prescribed condition. Figure 20 The parallax between the determination process of step S503 and the determination process of step S505 is shown. Figures 23 to 25 In the illustrated processing, a case where the object parallax includes both the first parallax and the second parallax is described.

[0178] In the process of step S604 , the control section 24 acquires the largest parallax among the first parallax and the second parallax associated with the u-coordinate of the search object stored in the memory 23 as the object parallax.

[0179] In the process of step S605, the control unit 24 calculates the v coordinate of the road surface corresponding to the object parallax. For example, the control unit 24 substitutes the object parallax into the value represented in Figure 7 The v coordinate of the road surface and the road surface parallax d obtained in the process of step S211 are shown as follows: r The approximate formula for the relationship between Figure 17 ) of the road parallax d r To calculate the v coordinate of the road surface. Figure 26 In the example, the u coordinate of the search target is set to coordinate (u1). In this case, the control unit 24 calculates coordinate (v1) as the v coordinate of the road surface.

[0180] In step S606, the control unit 24 determines whether there are coordinates associated with a parallax approximately equal to the object's parallax within a predetermined range along the negative v-axis from the calculated road surface's v-coordinate. In this disclosure, "parallax approximately equal to the object's parallax" includes both parallax identical to the object's parallax and parallax substantially identical to the object's parallax. In this disclosure, "parallax substantially identical to the object's parallax" means parallax that can be treated as the object's parallax and processed in image processing. For example, parallax within a range of ±10% of the object's parallax can be considered as parallax approximately equal to the object's parallax. Furthermore, the predetermined range in step S606 can be appropriately set based on the height of objects floating above the road surface. Examples of objects floating above the road surface include roadside trees, overpasses, and traffic lights. Such objects floating above the road surface are not detected by the object detection device 20. In other words, the height of such objects floating above the road surface is not calculated.

[0181] In step S606, if the control unit 24 does not determine that a coordinate associated with a parallax substantially equal to the target parallax exists within a predetermined range along the negative v-axis from the road surface's v-coordinate (step S606: No), the control unit 24 proceeds to step S607 without calculating the object's height. In step S607, the control unit 24 adds a scan-end flag to the target parallax acquired in step S604 and stored in memory 23. After executing step S607, the control unit 24 executes step S601 again. In the re-executed step S601, the control unit 24 determines whether a first parallax or a second parallax does not have a scan-end flag attached to the u-coordinate of the search object. In the re-executed step S604, the control unit 24 acquires the largest parallax between the first and second parallaxes that do not have a scan-end flag attached as the target parallax.

[0182] In the process of step S606, when the control unit 24 determines that there are coordinates associated with a parallax substantially equal to the object parallax within a predetermined range along the negative direction of the v-axis from the v-coordinate of the road surface (step S606: Yes), the process proceeds to step S608. In the process of step S608, the control unit 24 obtains as the first coordinates the coordinates associated with a parallax substantially equal to the object parallax within a predetermined range along the negative direction of the v-axis from the v-coordinate of the road surface. Figure 26In the example, the u coordinate of the search object is set to coordinate (u1). Furthermore, a parallax approximately equal to the object's parallax is associated with the coordinates (u1, v2) of the parallax pixel 71. In other words, the parallax pixel 71 is set to represent a parallax approximately equal to the object's parallax. Furthermore, the v coordinate of the road surface (v0) and the v coordinate of the parallax pixel 71 (v1) are set to be within a specified range. In this case, the control unit 24 acquires the coordinates (u1, v2) of the parallax pixel 71 as the first coordinate.

[0183] After executing the process of step S608, the control unit 24 determines whether a parallax substantially equal to the object parallax is associated with a coordinate obtained by subtracting 1 from the v coordinate of the first coordinate (step S609). Figure 26 In the example, the first coordinate is set to the coordinate (u1, v2) of the parallax pixel 71. The coordinate obtained by subtracting 1 from the v coordinate of the parallax pixel 71 is the coordinate (u1, v3) of the parallax pixel 72. The parallax pixel 72 is set to represent a parallax that is substantially equal to the object parallax. In other words, the parallax that is substantially equal to the object parallax is associated with the coordinate (u1, v3) of the parallax pixel 72. In this case, the control unit 24 determines that the parallax that is substantially equal to the object parallax is associated with the coordinate (u1, v3) of the parallax pixel 72 obtained by subtracting 1 from the v coordinate of the first coordinate (u1, v2).

[0184] In the process of step S609, if the control unit 24 determines that the parallax substantially equal to the object parallax is associated with the coordinate obtained by subtracting 1 from the v coordinate of the first coordinate (step S609: Yes), the process proceeds to step S610. In the process of step S610, the control unit 24 updates the coordinate obtained by subtracting 1 from the v coordinate of the first coordinate to the first coordinate. Figure 26 In the case where the first coordinate is the coordinate (u1, v2) of the parallax pixel 71, the first coordinate is updated to the coordinate (u1, v3) of the parallax pixel 72 by the process of step S610. After executing the process of step S610, the control unit 24 returns to the process of step S609. For example, the control unit 24 repeats the process of step S609 and the process of step S610 until the first coordinate is updated to the coordinate (u1, v3) of the parallax pixel 72. Figure 26 The coordinates (u1, v4) of the parallax pixel 73 shown are updated to the first coordinates. A parallax substantially equal to the object parallax is associated with the coordinates (u1, v4) of the parallax pixel 73.

[0185] In the process of step S609, when the control unit 24 determines that the parallax substantially equal to the object parallax is not associated with the coordinate obtained by subtracting 1 from the v coordinate of the first coordinate (step S609: No), it proceeds to step S609. Figure 24 The process of step S611 is shown. Figure 26In the example, the first coordinate is the coordinate (u1, v4) of parallax pixel 73. The coordinate obtained by subtracting 1 from the v coordinate of parallax pixel 73 is the coordinate (u1, v5) of parallax pixel 74. Parallax pixel 74 represents a parallax smaller than the parallax that is approximately equal to the object parallax. In other words, the parallax that is approximately equal to the object parallax is not associated with the coordinate (u1, v5) of parallax pixel 74. In this case, the control unit 24 determines that the parallax that is approximately equal to the object parallax is not associated with the coordinate (u1, v5) of parallax pixel 74 obtained by subtracting 1 from the v coordinate of the first coordinate (u1, v4).

[0186] In the process of step S611, the control unit 24 determines whether the v coordinate of the first coordinate is the minimum coordinate. When the v coordinate of the first coordinate is the minimum coordinate, it is when the control unit 24 finishes scanning the second parallax image along the negative direction of the v axis. When the v coordinate of the first coordinate is the minimum coordinate (step S611: yes), the control unit 24 enters the process of step S616. On the other hand, when the control unit 24 determines that the v coordinate of the first coordinate is not the minimum coordinate (step S611: no), the control unit 24 enters the process of step S612. For example, when the first coordinate is Figure 26 In the case of the coordinates (u1, v4) of the parallax pixel 73 shown, it is determined that the v coordinate (coordinate (v4)) of the first coordinate is not the minimum coordinate.

[0187] Before explaining the process of step S612, refer again to Figure 26 .exist Figure 26 In [ 1 ], the first coordinate is set to the coordinates (u1, v4) of parallax pixel 73. Parallax pixel 75 is located on the negative side of the v-axis relative to parallax pixel 73. Parallax pixel 75 is set to represent a parallax that is approximately equal to the object's parallax. Parallax pixel 73 and parallax pixel 75 are part of parallax image 42 corresponding to other vehicles that are the same object. However, due to parallax variations, even parallax pixels included in parallax images corresponding to the same object may be separated in the v-direction, as in parallax pixel 73 and parallax pixel 75.

[0188] Therefore, in the process of step S612, the control unit 24 determines whether there is a coordinate associated with a parallax substantially equal to the object parallax within a predetermined interval along the negative direction of the v-axis from the v-coordinate of the first coordinate. The predetermined interval can be appropriately set based on the height of the object to be detected by the object detection device 20. For example, the predetermined interval can be appropriately set based on the height of the back of the vehicle (for example, 80 cm). Figure 26, the interval in the v direction between the parallax pixel 73 and the parallax pixel 75 is within a predetermined interval. If the first coordinates are the coordinates (u1, v4) of the parallax pixel 73, the control unit 24 determines that the coordinates (u1, v6) of the parallax pixel 75 associated with a parallax substantially equal to the object parallax exist.

[0189] In the process of step S612, if the control unit 24 determines whether there are coordinates associated with a parallax substantially equal to the object parallax within a predetermined interval along the negative direction of the v-axis from the first coordinate (step S612: Yes), the process proceeds to step S613. On the other hand, if the control unit 24 determines that there are no coordinates associated with a parallax substantially equal to the object parallax within a predetermined interval along the negative direction of the v-axis from the first coordinate (step S612: No), the process proceeds to step S615.

[0190] The purpose of the process of step S613 is to determine whether the object parallax and the parallax substantially equal to the object parallax in the process of step S612 are parallaxes corresponding to the same object. Figure 27 and Figure 28 An example of the second parallax image will be described.

[0191] Figure 27 The second parallax image 80 is shown in FIG. The second parallax image 80 is based on Figure 28 The first image 90 is generated by the first image 90 shown in FIG. The first image 90 includes images 91 corresponding to other vehicles, images 92 corresponding to roadside trees, and images 93 corresponding to structures. Figure 27 As shown, the second parallax image 80 includes: Figure 28 The parallax image 81 corresponding to the image 91 shown, Figure 28 The parallax image 82 corresponding to the image 92 shown, and Figure 28 The parallax image 85 corresponding to the image 93 shown.

[0192] like Figure 27 As shown, the parallax image 81 includes a parallax pixel 83. The parallax image 82 includes a parallax pixel 84. The u coordinates of the parallax pixel 83 and the parallax pixel 84 are the same at the coordinate (u2). In addition, in the real space, the distance from the stereo camera 10 to Figure 28 The distance to the other vehicle corresponding to the image 91 shown and the distance from the stereo camera 10 to Figure 28 The distances to the roadside trees corresponding to the image 92 shown are roughly the same. Since these two distances are roughly the same, Figure 27 The parallax and Figure 27The parallax represented by the parallax pixels 84 of the parallax image 82 shown may be substantially equal. Figure 28 The other vehicles and Figure 28 The distances between the trees in the image 92 are close. Figure 27 The interval between the parallax pixels 83 of the parallax image 81 and the parallax pixels 84 of the parallax image 82 in the v direction shown is within the predetermined interval described above in the process of step S612 .

[0193] exist Figure 27 In the structure shown, the control unit 24 can obtain the disparity represented by the disparity pixel 83 and the disparity pixel 84 as the object disparity when the u coordinate of the search object is set to the coordinate (u2). In addition, the control unit 24 can update the coordinate of the disparity pixel 83 as the first coordinate. When the u coordinate of the search object is the coordinate (u2), it is desirable to calculate the height T1 corresponding to the disparity image 81 as the height of the object on the image. As an example, when the first coordinate is the coordinate of the disparity pixel 83, since the interval between the disparity pixel 83 and the disparity pixel 84 is within the above-mentioned prescribed interval, the control unit 24 considers updating the coordinate of the disparity pixel 84 as the first coordinate. In this example, the control unit 24 will continue to traverse and scan the disparity pixels of the disparity image 82 corresponding to the roadside tree. As a result, the control unit 24 detects the height of the disparity image 82, that is, the height T2 corresponding to the roadside tree, as the height of the object.

[0194] Here, multiple parallax pixels 86 exist between parallax pixels 83 and parallax pixels 84. Multiple parallax pixels 86 are included in parallax image 85. The structure corresponding to parallax image 85 is located farther from stereo camera 10 than the vehicle and the roadside trees. Therefore, the parallax represented by multiple parallax pixels 86 in parallax image 85 is smaller than the parallax represented by parallax pixels 83 and parallax pixels 84.

[0195] Therefore, in the process of step S613, the control unit 24 determines whether there are more than a predetermined number of coordinates associated with the third parallax between the first coordinate and the second coordinate. The second coordinate is a coordinate located within a predetermined interval along the negative direction of the v-axis from the first coordinate and is associated with a parallax that is substantially equal to the object parallax. For example, when the first coordinate is Figure 27 In the case of the coordinates of the parallax pixel 83 shown in FIG, the second coordinates become the coordinates of the parallax pixel 84. The third parallax is a parallax smaller than the object parallax. The third parallax can be set based on the parallax corresponding to the background. For example, the third parallax can be assumed to be Figure 27The predetermined number can be appropriately set based on the number of parallax pixels representing the third parallax included in the predetermined interval.

[0196] In the process of step S613, if the number of coordinates associated with the third parallax between the first coordinate and the second coordinate exceeds the prescribed number (step S613: Yes), the control unit 24 proceeds to the process of step S616. Figure 27 In the case of the coordinates of the parallax pixel 83 and the parallax pixel 84 shown in FIG, the control unit 24 determines that there are more than a predetermined number of parallax pixels 86 between the coordinates of the parallax pixel 83 and the coordinates of the parallax pixel 84. On the other hand, when the coordinates associated with the third parallax do not exceed the predetermined number between the first coordinate and the second coordinate (step S613: No), the control unit 24 proceeds to the process of step S614. For example, when the first coordinate and the second coordinate are Figure 26 In the case of the coordinates of the parallax pixel 73 and the coordinates of the parallax pixel 75 shown, it is determined that there are no parallax pixels 74 exceeding a predetermined number between the coordinates of the parallax pixel 73 and the coordinates of the parallax pixel 75 .

[0197] In the process of step S614, the control unit 24 updates the second coordinate to the first coordinate. For example, the control unit 24 updates the second coordinate to the first coordinate. Figure 26 When the coordinates of the parallax pixel 73 and the coordinates of the parallax pixel 75 are updated, the coordinates (u1, v6) of the parallax pixel 75 are updated to the first coordinates. After executing the process of step S614, the control unit 24 returns to Figure 23 For example, in step S609 Figure 26 In the illustrated configuration, the control unit 24 may repeatedly execute the processing from step S609 until the coordinates (u1, v7) of the parallax pixel 76 are updated to the first coordinates.

[0198] In the process of step S615, the control unit 24 determines whether there is a coordinate associated with a parallax substantially equal to the object parallax beyond a predetermined interval from the first coordinate in the negative direction of the v axis. Figure 26In the case of the coordinates (u1, v7) of the parallax pixel 76 shown, it is determined that there are no coordinates associated with a parallax substantially equal to the object parallax beyond the prescribed interval from the coordinates (u1, v7) along the negative direction of the v-axis. When there are no coordinates associated with a parallax substantially equal to the object parallax beyond the prescribed interval from the first coordinate along the negative direction of the v-axis (step S615: No), the control unit 24 proceeds to the processing of step S616. On the other hand, when it is determined that there are coordinates associated with a parallax substantially equal to the object parallax beyond the prescribed interval from the first coordinate along the negative direction of the v-axis (step S615: Yes), the control unit 24 proceeds to the processing of step S616. Figure 25 The process of step S618 is shown.

[0199] In the process of step S616, the control unit 24 calculates the height of the object on the image by subtracting the v coordinate of the road surface from the v coordinate of the first coordinate. Figure 26 In the case of the coordinates (u1, v7) of the parallax pixel 76 shown, the control unit 24 subtracts the v coordinate of the road surface (coordinate (v1)) from the v coordinate of the parallax pixel 76 (coordinate (v7)) to calculate the height of the object. Figure 27 In the case of the coordinates of the parallax pixel 83 shown in FIG, the control unit 24 subtracts the v coordinate of the road surface (coordinate (v8)) from the v coordinate of the parallax pixel 83 to calculate the height T1 of the object. Figure 27 As shown, even when the parallax image 81 of the other vehicle and the parallax image 82 of the roadside tree are close to each other, the height of the other vehicle can be calculated with high accuracy. In step S616, the control unit 24 associates the calculated height of the object on the image with the first coordinate and stores it in the memory 23.

[0200] In step S616, the control unit 24 may convert the calculated height of the object in the image into the height of the object in real space. If the height of the object in real space is less than the minimum height of the object to be detected, the control unit 24 may discard the calculated height information of the object. For example, if the minimum height of the object to be detected is 50 cm, the control unit 24 may discard the calculated height information of the object if the height of the object in real space is 40 cm less.

[0201] After executing the process of step S616, the control unit 24 executes the process of step S617. In the process of step S617, the control unit 24 adds a scan end mark to the object parallax. After executing the process of step S617, the control unit 24 returns to Figure 23 The process of step S601 is shown.

[0202] Before explaining the process of step S618, refer to Figure 29 An example of the second parallax image is described below. Figure 29 The second parallax image 100 is shown in FIG. Figure 29 A portion of the second parallax image 100 is shown in FIG. The parallax image 101 is based on Figure 30 The parallax image is generated by comparing the first image 106 shown. The first image 106 includes an image 107 corresponding to the back of the truck.

[0203] like Figure 29 As shown, the second parallax image 100 includes a parallax image 101. The parallax image 101 and Figure 30 The parallax image 101 includes parallax pixels 102, 103, 104, and 105. The parallax pixels 103 and 105 correspond to each other. Figure 30 The lower image 107a of the image 107 shown in FIG. The parallax pixels 104 and 105 correspond to Figure 30 The parallax pixel 103 corresponds to the upper image 107b of the illustrated image 107. The parallax represented by the parallax pixel 103, the parallax represented by the parallax pixel 104, and the parallax represented by the parallax pixel 105 are parallaxes corresponding to the same object, the truck, and are therefore substantially equal.

[0204] like Figure 29 As shown, multiple disparity pixels 102 are located in the center of disparity image 101. Disparity is not calculated for these disparity pixels 102. In other words, these disparity pixels 102 do not contain disparity information. Typically, the number of features in the stereoscopic image is low in the center of the back of a truck. Because the number of features in the stereoscopic image is low in the center of the back of a truck, the aforementioned matching process for calculating disparity may not calculate disparity for these disparity pixels 102.

[0205] exist Figure 29 In the illustrated configuration, as an example, consider an example where the control unit 24 sets the u coordinate of the search object to the coordinate (u3) and scans in the v direction. In this example, it is desired to calculate the height T3 corresponding to the parallax image 101, that is, the height of the truck as the height of the object. In this example, the control unit 24 can obtain the parallax represented by the parallax pixel 103 as the object parallax. In addition, the control unit 24 can set the coordinate (u3, v 10 ) is updated to the first coordinate. In order to calculate the height T3, the coordinates (u3, v 11 ) is updated to the first coordinate. In order to update the coordinates (u3, v 11) is updated to the first coordinate, the prescribed interval in the above step S612 needs to be increased to be larger than the v coordinate of the parallax pixel 104 (coordinate (v 11 )) and the v coordinate of the parallax pixel 103 (coordinate (v 10 However, if the predetermined interval in the process of the above-mentioned step S612 is increased, the possibility of miscalculating the height of the object will increase.

[0206] Therefore, the control unit 24 pre-calculates the Figure 29 The height T4 shown is used as the first candidate height, and the Figure 29 The height T3 shown is used as the second candidate height. Figure 5 In step S107, it is determined which of the first and second candidate heights is to be used as the object's height. This configuration eliminates the need to increase the predetermined interval in step S612. This avoids increasing the predetermined interval in step S612, thereby reducing the likelihood of miscalculating the object's height.

[0207] In the process of step S618, the control unit 24 obtains candidate coordinates. Candidate coordinates are coordinates that exist beyond a predetermined interval in the negative direction of the v-axis from the first coordinate and are associated with a parallax that is substantially equal to the object parallax. For example, the control unit 24 obtains candidate coordinates when the first coordinate is Figure 29 The coordinates (u3, v 10 ), the coordinates (u3, v 11 ) as candidate coordinates.

[0208] In the process of step S619, the control unit 24 calculates the first candidate height by subtracting the v coordinate of the road surface from the v coordinate of the first coordinate. Figure 29 The coordinates (u3, v 10 ), from the v coordinate (coordinate (v 10 )) minus the v coordinate of the road surface (coordinate (v9)) to calculate the first candidate height T4. In the process of step S619, the control unit 24 associates the calculated first candidate height with the first coordinate and stores it in the memory 23. For example, the control unit 24 Figure 29 The coordinates of the first candidate height T4 and the first coordinate disparity pixel 103 (u3, v 10 ) are associated and stored in memory 23.

[0209] In the process of step S620, the control unit 24 determines whether the parallax that is substantially equal to the object parallax is associated with the coordinates obtained by subtracting 1 from the v coordinate of the candidate coordinates. When the control unit 24 determines that the parallax that is substantially equal to the object parallax is associated with the coordinates obtained by subtracting 1 from the v coordinate of the candidate coordinates (step S620: Yes), the control unit 24 proceeds to the process of step S621. In the process of step S621, the control unit 24 updates the coordinates obtained by subtracting 1 from the v coordinate of the candidate coordinates to the candidate coordinates. After executing the process of step S621, the control unit 24 returns to the process of step S620. For example, in Figure 29 In the structure shown, the control unit 24 repeatedly executes the process of step S620 and the process of step S621 until the coordinates (u3, v 12 ) is updated as the candidate coordinates. On the other hand, when the control unit 24 determines that the parallax substantially equal to the object parallax is not associated with the coordinate obtained by subtracting 1 from the v coordinate of the candidate coordinates (step S620: No), the process proceeds to step S622. For example, when the candidate coordinates are Figure 29 The coordinates (u3, v 12 ), it is determined that the parallax that is substantially equal to the object parallax is not associated with the coordinates obtained by subtracting 1 from the v coordinate of the candidate coordinates.

[0210] In the process of step S622, the control unit 24 calculates the second candidate height by subtracting the v coordinate of the road surface from the v coordinate of the candidate coordinate. Figure 29 The coordinates (u3, v 12 ), from the v coordinate (coordinate (v 12 )) minus the v coordinate of the road surface (coordinate (v9)) to calculate the second candidate height T3. In the process of step S622, the control unit 24 associates the calculated second candidate height with the candidate coordinates and stores them in the memory 23. For example, the control unit 24 Figure 29 The coordinates of the second candidate height T3 and the disparity pixel 105 of the candidate coordinates (u3, v 12 ) is associated and stored in the memory 23. After executing the process of step S622, the control unit 24 proceeds to the process of step S623.

[0211] In the process of step S623, the control unit 24 adds a scan end mark to the object parallax acquired in the process of step S604 stored in the memory 23. After executing the process of step S623, the control unit 24 returns to Figure 23 The process of step S601 is shown.

[0212] In the process of step S106, the control unit 24 performs parallel object detection processing. The details of the process of step S106 are as follows: Figure 31 As shown in the flowchart shown.

[0213] In the process of step S701, the control unit 24 generates or acquires a UD map. The UD map is also called "U-disparity space" and "ud coordinate space". The UD map is obtained by associating the object disparity with a two-dimensional coordinate consisting of the u direction and the d direction corresponding to the magnitude of the disparity. The object disparity may be only the first disparity detected as the disparity corresponding to the object. Alternatively, the object disparity may include both the first disparity and the second disparity. Hereinafter, the object disparity is assumed to include the first disparity and the second disparity. The control unit 24 may acquire the first disparity and the second disparity stored in the memory 23 and generate the UD map. Alternatively, the control unit 24 may acquire the UD map from the outside via the acquisition unit 21.

[0214] Figure 32 The UD diagram 110 is shown in FIG. The horizontal axis of the UD diagram 110 corresponds to the u axis. The vertical axis of the UD diagram 110 corresponds to the d axis representing the magnitude of the parallax. The coordinate system composed of the u coordinate and the d coordinate is also called the "ud coordinate system". In the UD diagram 110, the direction Figure 32 The lower left corner of the paper becomes the origin (0, 0) of the ud coordinate system. Figure 32 The plot shown is made using Figure 20 The coordinate points associated with the first and second parallaxes detected by the process shown are also referred to as "points". The UD map 110 includes point groups 111, 112, 113, 114, and 115. The UD map 110 is based on the Figure 33 The second parallax image is generated from the first image 120 shown.

[0215] like Figure 33 As shown, the first image 120 includes: an image 121 corresponding to the guardrail, images 122 and 123 corresponding to other vehicles, an image 124 corresponding to pedestrians, and an image 125 corresponding to the sidewall. The first and second parallaxes detected from each of the images 121 to 125 are Figure 32 Each point group in the point groups 111 to 115 shown corresponds to an image 121 and an image 125 that are parallel objects. Figure 32 The point group 111 and the point group 115 shown may be the parallax corresponding to the parallel objects. One purpose of the parallel object detection process is to detect Figure 32 Parallax corresponding to parallel objects such as point group 111 and point group 115 are shown.

[0216] In this embodiment, Figure 32The parallax corresponding to parallel objects such as the point group 111 and the point group 115 shown in FIG. 1 is detected using Hough transform in the process of step S704 described later. Figure 32 As shown, point group 112 and point group 113 are located near point group 111 and point group 115. When point group 112 and point group 113 are located near point group 111 and point group 115, the accuracy of detecting point group 111 and point group 115 by using Hough transform in the process of step S704 described later will be reduced. Therefore, by using the process of step S702, the control unit 24 determines whether there is a point group located near point group 111 and point group 115. Figure 32 Point groups 112 and 113 are shown as being near point groups 111 and 115 .

[0217] Specifically, in the process of step S702, the control unit 24 determines whether there is a point group substantially parallel to the u direction. Figure 32 The point groups 112 and 113 shown are located near the point groups 111 and 115. Figure 33 The other vehicles include a portion that is substantially parallel to the width direction of the road surface. Since the other vehicles include a portion that is substantially parallel to the width direction of the road surface, the point group 112 and the point group 113 are as shown in FIG. Figure 32 Therefore, we can determine whether there is a point group roughly parallel to the u direction. Figure 32 Point group 112 and point group 113 are shown.

[0218] As an example of the process of step S702, first, the control unit 24 scans along the u direction of the UD map. Figure 32 The UD diagram 110 shown is scanned from the negative direction of the u-axis to the positive direction of the u-axis. The control unit 24 determines whether there are points that are continuously arranged in the u-direction within a specified range while scanning along the u-direction of the UD diagram. The specified range is based on the image along the vehicle (for example, Figure 33 The control unit 24 determines that a point group approximately parallel to the u direction exists when it determines that points continuously arranged in a prescribed range along the u direction exist. The control unit 24 detects the parallax points continuously arranged in a prescribed interval along the u direction as a point group approximately parallel to the u direction. For example, the control unit 24 Figure 32 The point groups 112 and 113 shown are detected as point groups substantially parallel to the u direction.

[0219] In the process of step S702, if the control unit 24 determines that a point cluster substantially parallel to the u direction exists (step S702: Yes), the process proceeds to step S703. On the other hand, if the control unit 24 determines that a point cluster substantially parallel to the u direction does not exist (step S702: No), the process proceeds to step S704.

[0220] In step S703, the control unit 24 removes the detected point clusters that are substantially parallel to the u direction from the UD map. Instead of removing the point clusters that are substantially parallel to the u direction from the UD map, the control unit 24 may simply exclude the point clusters that are substantially parallel to the u direction from the coordinate points of the UD map by using the Hough transform in step S704 described later. Figure 34 The UD diagram 110 is shown with the point group roughly parallel to the u direction removed. Figure 34 In the UD diagram 110 shown, Figure 32 Point group 112 and point group 113 are shown removed.

[0221] In the processing of step S704, the control unit 24 uses the Hough transform on the points included in the UD map to detect a straight line. Here, the UD map can also be converted into a coordinate system of the actual space composed of xz coordinates, and the Hough transform can be used to detect a straight line on the converted coordinate system of the actual space. If the baseline length B between the first camera 11 and the second camera 12 is short, there is a situation where the intervals between the coordinate points included in the coordinate system of the actual space are farther apart than the intervals between the coordinate points included in the UD map. If the intervals between the coordinate points are far apart, there is a situation where the Hough transform cannot detect a straight line with high precision. In this embodiment, by using the Hough transform on the UD map, a straight line can be detected with high precision even when the baseline length B between the first camera 11 and the second camera 12 is short.

[0222] Reference Figure 35 and Figure 36 An example of the processing of step S704 will be described.

[0223] exist Figure 35 A portion of the UD graph is shown in FIG. Points 131, 132, 133, and 134 are coordinate points on the UD graph. The Hough transform is explained using point 131 as an example. The uv coordinate of point 131 on the UD graph is the coordinate (u 131 , d 131 ). The straight line L1 passing through the point 131 can be defined infinitely. For example, the length of the normal line of the straight line L1-1 from the origin (0, 0) of the uv coordinate is length r. This normal line is inclined at an angle θ from the u-axis toward the positive direction of the d-axis. The control unit 24 obtains the following equation (3) as a general formula for the straight line L1 passing through the point 131 by using the length r and the angle θ as variables.

[0224] r=u131 ×cosθ+d 131 ×sinθ (3)

[0225] The control unit 24 projects the equation (Equation (3)) of the straight line L1 onto Figure 36 The rθ plane of the Hough space is shown. Figure 36 The horizontal axis of the rθ plane shown is the r-axis. Figure 36 The vertical axis of the rθ plane shown is the θ axis. Figure 36 The curve 131L shown is a curve expressed by equation (3). The curve 131L is expressed as a sine curve in the rθ plane. Similar to point 131, the control unit 24 obtains Figure 35 The formula of the straight line passing through points 132 to 134 is shown in FIG. Similar to point 131, the control unit 24 projects the obtained formula of the straight line passing through points 132 to 134 onto FIG. Figure 36 Shown as the rθ plane of the Hough space. Figure 36 The curves 132L to 134L shown correspond to the obtained Figure 35 The straight line from point 132 to point 134 is shown. Figure 36 As shown, curves 131L to 134L can be at point P L The control unit 24 obtains point P L The rθ coordinate (coordinate (θ L , r L )). The control unit 24 is based on point P L The coordinates (θ L , r L ) to detect the Figure 35 The control unit 24 detects the following equation (4) as the equation of the straight line between points 131 to 134. Figure 35 The straight line from point 131 to point 134 is shown.

[0226] r L =u×cosθ L +d×sinθ L (4)

[0227] By executing the process of step S704, the control unit 24 can detect Figure 34 The straight line 111L corresponding to the point group 111 and the straight line 115L corresponding to the point group 115 are shown.

[0228] However, if Figure 33 As shown, the images 121 and 125 corresponding to the parallel objects are directed toward the vanishing point 120. VP By moving the image 121 and the image 125 toward the vanishing point 120 VP Extension, such as Figure 34 As shown, point group 111 and point group 115 also move toward vanishing point 120.VP The corresponding vanishing point is 110 VP By extending the point group 111 and the point group 115 to the vanishing point 110 VP By extending, the straight line 111L and the straight line 115L can also be extended to the vanishing point 110 VP extend.

[0229] Therefore, in the process of step S704, when the control unit 24 obtains the formula of the straight line passing through the point on the UD graph, the control unit 24 may obtain the formula of the straight line passing through the prescribed range based on the vanishing point among the straight lines passing through the point. Figure 35 When the formula of the straight line L1 passing through the point 131 is calculated, the formula of the straight line L1 passing through the point 131 and the predetermined range Δu can be obtained. VP The formula of the straight line. Specified range Δu VP It is the range based on the vanishing point. VP Point 135 is included. Point 135 may be a vanishing point when the moving object 30 is moving in a straight line. Since the parallax at infinity is zero, the d coordinate of point 135 may be zero. When the moving object 30 is moving in a straight line, since the u coordinate of the vanishing point is half of the maximum u coordinate, the u coordinate of point 135 (coordinate u VP ) can be half of the maximum coordinate of u coordinate. VP The vehicle 30 can be driven on a curve based on the Figure 34 The vanishing point 110 is shown VP The offset relative to point 135 is appropriately set. Figure 36 As shown, in the rθ plane of the Hough space, the range of the θ axis for plotting the curves 131L to 134L can be limited to the range Δθ VP By limiting the range of the θ axis where the curves 131L to 134L are drawn to the range Δθ VP , the amount of computation required for the Hough transform can be reduced. By reducing the amount of computation required for the Hough transform, the processing speed of the Hough transform can be increased.

[0230] In the process of step S705, the control unit 24 determines whether the length of the straight line detected by the process of step S704 exceeds the prescribed length. The prescribed length can be appropriately set based on the length of the structure set along the road surface, that is, the length of the parallel object. When the control unit 24 determines that the length of the straight line is less than the prescribed length (step S705: No), it returns to the control unit 24. Figure 5On the other hand, when the control unit 24 determines that the length of the straight line exceeds the prescribed length (step S705: Yes), the control unit 24 adds a parallel object mark to the point group corresponding to the straight line (step S706). For example, the control unit 24 adds a parallel object mark to the point group corresponding to the straight line (step S706). Figure 34 After executing the process of step S706, the control unit 24 enters Figure 5 The process of step S107 is shown.

[0231] In the process of step S107, the control unit 24 performs the recovery process. The details of the process of step S107 are as follows: Figure 37 As shown in the flowchart shown.

[0232] In the process of step S801, the control unit 24 and Figure 31 The processing of step S701 shown in FIG. 1 is the same or similar to generating or acquiring a UD map. Figure 31 In the case where a UD map is generated in the process of step S701, the control unit 24 may obtain Figure 31 The UD map generated in the process of step S701 is shown.

[0233] Figure 38 UD diagram 140 is shown in FIG. UD diagram 140 is Figure 34 The UD diagram 110 is a partially enlarged diagram. The horizontal axis of the UD diagram 140 corresponds to the u axis. The vertical axis of the UD diagram 140 corresponds to the d axis indicating the magnitude of the parallax. Figure 38 , the UD image 140 is represented as an image. The pixels of the UD image 140 represent parallax. The pixels marked with hatching are pixels representing the first parallax. The pixels representing the first parallax, that is, the coordinates associated with the first parallax are also called "first parallax points". The pixels marked with dots are pixels representing the second parallax. The pixels representing the second parallax, that is, the coordinates associated with the second parallax are also called "second parallax points". The UD image 140 includes a first parallax point 141, a first parallax point 142, and a second parallax point 143. The UD image 140 is generated based on the second parallax image including the parallax image corresponding to the back of the vehicle. In the central part of the back of the vehicle, due to the presence of the rear glass, etc., the feature quantity on the stereoscopic image is small. In the central part of the back of the vehicle, due to the small feature quantity on the stereoscopic image, compared with the parallax of other parts, such as Figure 19 As shown in the parallax image 44, there are cases where less parallax information can be obtained. Figure 38 As shown, there is a second parallax point 143 between the first parallax point 141 and the first parallax point 142 .

[0234] In the processing of step S802, the control unit 24 determines whether the second parallax points sandwiched by the first parallax points in the u direction of the UD image exist beyond the prescribed range. For example, the control unit 24 scans from the negative direction side of the u-axis of the UD image to the positive direction side of the u-axis. The control unit 24 determines whether the second parallax points sandwiched by the first parallax points in the u direction of the UD image exist beyond the prescribed range by scanning along the u direction. The prescribed range can be appropriately set based on the width of the back of the vehicle in the actual space (for example, 1m). In the case where the second parallax points sandwiched by the first parallax points exist beyond the prescribed range, there is a high possibility that these second parallax points are parallaxes corresponding to different objects, such as different vehicles running in parallel. In contrast, in the case where the second parallax points sandwiched by the first parallax points exist within the prescribed range, there is a high possibility that these second parallax points are parallaxes corresponding to the same object. In Figure 38 In the u direction of the UD image 140, a second parallax point 143, which is sandwiched between the first parallax point 141 and the first parallax point 142, exists within a predetermined range. In the u direction of the UD image 140, the control unit 24 does not determine that the second parallax point 143, which is sandwiched between the first parallax point 141 and the first parallax point 142, exists beyond the predetermined range.

[0235] In step S802, if the control unit 24 determines that the second parallax points sandwiched between the first parallax points in the u direction of the UD image exceed the predetermined range (step S802: Yes), the process proceeds to step S805. On the other hand, if the control unit 24 does not determine that the second parallax points sandwiched between the first parallax points in the u direction of the UD image exceed the predetermined range (step S802: No), the process proceeds to step S803.

[0236] In the process of step S803, the control unit 24 obtains the first parallax associated with each of the two first parallax points sandwiching the second parallax point. Figure 38 The first parallax associated with the first parallax point 141 and the first parallax associated with the first parallax point 142 are shown. Then, in the process of step S802, the control unit 24 determines whether the difference in height in the real space corresponding to the two first parallaxes is within a predetermined height. For example, the control unit 24 determines Figure 38 Is the difference between the height in real space corresponding to the first parallax of the first parallax point 141 and the height in real space corresponding to the first parallax of the first parallax point 142 below a predetermined height? The predetermined height can be appropriately set based on the height of the vehicle in real space (e.g., 1 meter).

[0237] In step S803, if the control unit 24 determines that the difference in real-world heights corresponding to the two acquired first parallaxes is within a specified range (step S803: Yes), the process proceeds to step S804. In step S804, the control unit 24 applies a restoration flag to the second parallax points that are sandwiched between the first parallax points and are outside the specified range. As described above, the second parallax points are the ud coordinates representing the second parallax. In other words, the process of step S804 can also be referred to as applying a restoration flag to the second parallax and the u coordinate associated with it. Figure 39 UD graph 140 with restoration markings added is shown in FIG. Figure 39 The pixels shown with thick hatching are second disparity points to which a restoration mark is attached.

[0238] In the process of step S803 , when the control unit 24 determines that the difference in height in the real space corresponding to the two acquired first parallaxes exceeds a predetermined height (step S803 : No), the process proceeds to step S805 .

[0239] The processing after step S805 is to determine the Figure 25 The first candidate height calculated in the process of step S619 shown in FIG. Figure 25 Before explaining the processing after step S805, refer to Figure 40 and Figure 41 An example of the second parallax image will be described.

[0240] Figure 40 The second parallax image 150 is shown in FIG. Figure 41 The second parallax image 160 is shown in FIG. Figure 40 and Figure 41 FIG2 shows a portion of the second parallax image 150 and the second parallax image 160. In the second parallax image 150 and the second parallax image 160, white parallax pixels are pixels that do not contain parallax information. Parallax pixels with thick hatching are pixels that contain parallax information. Parallax pixels with thin hatching are pixels for Figure 25 The coordinates of the disparity pixels marked with thin hatching are in the second candidate height calculated in the process of step S622. Figure 25 The disparity pixels marked with a large dot are used to associate the candidate coordinates with the second candidate height in the process of step S622. Figure 25 The coordinates of the disparity pixels marked with a large dot are in Figure 25The first candidate height is associated with the first coordinate in the process of step S619 shown in FIG. Figure 24 The coordinates of the disparity pixels marked with small dots are in Figure 24 In the process of step S616 shown, the height of the object is associated as the first coordinate.

[0241] and Figure 29 The second parallax image 100 shown is similarly Figure 40 The second parallax image 150 is shown based on Figure 30 The image 107 of the back of the truck is generated. The white parallax pixels are located in the center of the second parallax image 150. In other words, the parallax of the parallax pixels in the center is not calculated in the second parallax image 150. The parallax of the parallax pixels surrounding the center is calculated in the second parallax image 150.

[0242] Figure 40 Second disparity image 150 shown includes disparity pixels 151, 152, 153, and 154. Disparity pixel 151 is a disparity pixel used to calculate the second candidate height. The coordinates of disparity pixel 151 are associated with the second candidate height as candidate coordinates. Disparity pixels 152 and 153 are disparity pixels used to calculate the height of an object. The coordinates of disparity pixels 152 and 153 are each associated with the height of the object as a first coordinate. Disparity pixel 154 is a disparity pixel used to calculate the first candidate height. The coordinates of disparity pixel 154 are associated with the first candidate height as a first coordinate.

[0243] and Figure 29 Similar to the parallax pixel 105 shown, the parallax pixel 151 is Figure 30 The upper image 107b corresponds to the image 107 of the back of the truck shown. Figure 29 Similar to the parallax pixel 103 shown, the parallax pixel 154 is Figure 30 The lower image 107a of the image 107 shows the back of the truck. Figure 40 In the structure shown, among the second candidate height based on the parallax pixel 151 and the first candidate height based on the parallax pixel 154, the second candidate height based on the parallax pixel 151 needs to be obtained as the height of the object, that is, Figure 30 Height of truck shown.

[0244] Figure 41 The second parallax image 160 is shown based on Figure 42A parallax image is generated based on the first image 170 shown. First image 170 includes image 171 of the upper portion of another vehicle and image 172 corresponding to a roadside tree. The parallax generated by image 171 and image 172 is approximately equal. Image 172 includes partial image 172a and partial image 172b. Partial image 172a and vehicle image 171 are located in the center of first image 170 in the u direction. The u coordinate of partial image 172a is the same as a portion of the u coordinate of image 171 of the other vehicle. Partial image 172a is located on the negative side of the v axis compared to image 171 of the other vehicle. Partial image 172b is located on the negative side of the u axis compared to partial image 172a and image 171.

[0245] Figure 41 The second parallax image 160 shown includes parallax pixels 161, 162, and 163. Parallax pixel 161 is a parallax pixel used for calculating the second candidate height. Figure 42 The height of the partial image 172a of the roadside tree is used as the second candidate height. The coordinates of the parallax pixel 161 are associated with the second candidate height as candidate coordinates. The parallax pixel 162 is a parallax pixel used for calculating the height of the object. Figure 42 The height of the partial image 172b of the roadside tree is used as the height of the object. The coordinates of the parallax pixel 162 are associated with the height of the object as candidate coordinates. The parallax pixel 163 is the parallax pixel used for calculating the first candidate height. Figure 42 The height of the vehicle image 171 is shown as a first candidate height. The coordinates of the parallax pixel 163 are associated with the first candidate height as first coordinates.

[0246] exist Figure 41 In the structure shown, among the first candidate height based on the parallax pixel 163 and the second candidate height based on the parallax pixel 161, the first candidate height based on the parallax pixel 163 needs to be obtained as the height of the object, that is, Figure 42 The height of the image 171 of the vehicle is shown.

[0247] In the processing of steps S805 to S807 , the control unit 24 determines which of the first candidate height and the second candidate height to acquire as the height of the object based on the parallax associated with the two coordinates sandwiching the candidate coordinates.

[0248] Specifically, in the process of step S805, the control unit 24 scans the second parallax image along the u direction to determine whether there are two first coordinates sandwiching the candidate coordinates in the u direction. Figure 23As described above, a parallax substantially equal to the target parallax is associated with the first coordinate. That is, the process of step S805 can be referred to as a process of determining whether the parallaxes associated with the two coordinates sandwiching the candidate coordinate in the u direction are target parallaxes.

[0249] For example, in Figure 40 In the u direction of the second parallax image 150 shown, the candidate coordinates of the parallax pixel 151 are sandwiched between the first coordinates of the parallax pixel 152 and the first coordinates of the parallax pixel 153. Figure 40 In the illustrated configuration, the control unit 24 determines that the first coordinates of the parallax pixel 152 and the first coordinates of the parallax pixel 153 exist, sandwiching the candidate coordinates of the parallax pixel 151 .

[0250] For example, in Figure 41 In the u direction of the second parallax image 160 shown, the first coordinate of the parallax pixel 162 is located on the negative side of the u axis of the candidate coordinates of the parallax pixel 161. On the other hand, the first coordinate is not located on the positive side of the u axis of the candidate coordinates of the parallax pixel 161. Figure 41 In the illustrated configuration, the control unit 24 determines that there are no two first coordinates sandwiching the candidate coordinates of the parallax pixel 161 .

[0251] In the process of step S805, if the control unit 24 determines that two first coordinates sandwiching the candidate coordinates in the u direction of the second parallax image exist (step S805: Yes), the process proceeds to step S806. On the other hand, if the control unit 24 determines that two first coordinates sandwiching the candidate coordinates in the u direction of the second parallax image do not exist (step S805: No), the process proceeds to step S807.

[0252] In the process of step S806, the control unit 24 determines to obtain the second candidate height of the first candidate height and the second candidate height as the height of the object. When the control unit 24 determines that the parallax associated with the two coordinates sandwiching the candidate coordinates is the object parallax through the process of steps S805 and S806, it determines to obtain the second candidate height as the height of the object. For example, the control unit 24 determines that based on Figure 40 Of the first candidate height based on the coordinates of the parallax pixel 154 and the second candidate height based on the coordinates of the parallax pixel 151 , the second candidate height based on the coordinates of the parallax pixel 151 is acquired as the height of the object. Figure 43 Shown in Figure 40 The parallax pixels used for determining the height of the object in the second parallax image 150 are shown. Figure 43 In , the height of the object is calculated based on the coordinates of the disparity pixels with thin shadow lines. Figure 43As shown, among the second candidate height based on the parallax pixel 151 and the first candidate height based on the parallax pixel 154, the second candidate height based on the parallax pixel 151 is obtained as the height of the object, that is, Figure 30 Height of truck shown.

[0253] In the process of step S807, the control unit 24 determines to obtain the first candidate height of the first candidate height and the second candidate height as the height of the object. If the control unit 24 does not determine that the parallax associated with the two coordinates sandwiching the candidate coordinates is the object parallax through the process of steps S805 and S807, it determines to obtain the first candidate height as the height of the object. For example, the control unit 24 determines that Figure 41 Of the first candidate height based on the coordinates of the parallax pixel 163 and the second candidate height based on the coordinates of the parallax pixel 161 shown, the first candidate height based on the coordinates of the parallax pixel 163 is acquired as the height of the object. Figure 44 Shown in Figure 41 The parallax pixels used for determining the height of the object in the second parallax image 160 are shown. Figure 44 In , the height of the object is calculated based on the coordinates of the parallax pixels with thin shadow lines. Figure 44 As shown, among the first candidate height based on the parallax pixel 163 and the second candidate height based on the parallax pixel 161, the first candidate height based on the parallax pixel 163 is obtained as the height of the object, that is, Figure 42 The height of the image 171 of the vehicle is shown.

[0254] After executing steps S806 and S807, the control unit 24 enters Figure 5 The process of step S108 is shown.

[0255] In the process of step S108, the control unit 24 performs a process of determining the representative parallax. The details of the process of step S108 are described in detail. Figure 45 The flow chart shown is shown.

[0256] In the process of step S901, the control unit 24 and Figure 31 The processing of step S701 shown in FIG. 1 generates a UD graph in the same or similar manner. Figure 31 In the case of producing a UD diagram in the process of step S701 shown in FIG. Figure 31 The UD map generated in the process of step S701 is shown.

[0257] In the process of step S902, the control unit 24 selects the first parallax and the u coordinate of the UD map. Figure 37 The control unit 24 obtains the representative parallax from the second parallax to which the restoration flag is added in the process of step S804. Figure 31 The process of step S706 shown above obtains a representative disparity from the first disparity to which the parallel object flag is not attached and the second disparity to which the parallel object flag is not attached but to which the restoration flag is attached.

[0258] Here, when the distance from the stereo camera 10 to an object is close, the number of pixels occupied by the object in the stereo image can be greater than when the distance from the stereo camera 10 is long. Furthermore, the farther the object is from the stereo camera 10, the lower the accuracy of the parallax detection corresponding to the object due to the influence of noise and other factors. In other words, the closer the object is to the stereo camera 10, the higher the accuracy of the parallax detection corresponding to the object.

[0259] Therefore, in the process of step S902, the control unit 24 obtains a representative disparity from the first disparity and the second disparity with the restoration flag added at each u-coordinate in the UD map as the disparity corresponding to an object located closer to the stereo camera 10, that is, the maximum disparity. Furthermore, the control unit 24 may obtain, as the representative disparity, a disparity corresponding to an object located closer to the stereo camera 10 and greater than the minimum height of objects to be detected by the object detection device 20 (e.g., 50 cm).

[0260] Figure 46 An example of the representative disparity obtained corresponding to the u direction is shown in FIG. Figure 46 In the example shown, the deviation of the representative parallax with respect to each u-coordinate is large. If the deviation of the representative parallax with respect to each u-coordinate is large, the amount of calculation required for the grouping process in step S109 may increase.

[0261] Therefore, in the processing of step S903, the control unit 24 may average the representative disparities within a specified range. By averaging the representative disparities within the specified range, the computational complexity of the grouping process in the processing of step S109 can be reduced. The specified range can be appropriately set taking into account the computational load of the processing of step S109, described later, and the like. During the averaging, the control unit 24 may obtain the representative disparity of the central value within the specified range. The control unit 24 may remove the representative disparities that are offset by a specified percentage (e.g., 5%) relative to the extracted central value within the specified range and calculate the average value of the representative disparities. Figure 47 An example of the averaged representative parallax corresponding to the u direction is shown in FIG. After executing the process of step S903, the control unit 24 enters Figure 5 The process of step S109 is shown.

[0262] The control unit 24 can execute Figure 5After the processing of steps S104 to S108 shown in the figure, that is, the third processing, for each predetermined range of horizontal coordinates (u coordinates) including one or more coordinates, the parallax d is represented. e The height information of the object is associated with the object and stored in the memory 23. Figure 48 As shown, the plurality of representative disparities d stored in the memory 23 e It can be expressed as the distribution of a point group in a two-dimensional space (ud coordinate space) with the u coordinate and the parallax d as the horizontal axis and the vertical axis respectively.

[0263] In the process of step S109, the control unit 24 sets the representative parallax d e The information is converted into a coordinate system of the actual space composed of xz coordinates, and the parallax d is extracted. e The cluster (group) of , thereby performing the process of detecting the object (the fourth process). Figure 49 and Figure 50 An example of the processing of step S109 will be described.

[0264] Figure 49 This is an example of a structure in real space. Figure 49 The vehicle 30 and other vehicles 42A are installed with the object detection system 1 and are traveling on a road surface 41A. Figure 49 In FIG, the moving object 30 is a vehicle.

[0265] The control unit 24 of the object detection device 20 mounted on the moving body 30 Figure 48 The disparity d of multiple objects in the ud coordinate space shown e Convert to Figure 50 The point group of the actual space (xz coordinate space) shown in . Figure 50 In the ud coordinate space, the representative disparity d is represented e Each point is displayed as a point in the xz coordinate space. The control unit 24 extracts a cluster of point groups based on the distribution of the point group. The control unit 24 collects and extracts a plurality of close points as a cluster of point groups according to a predetermined condition. The cluster of point groups represents the parallax d. e The cluster (group).

[0266] When an object has a surface parallel to the baseline length direction of the stereo camera 10, the point group is arranged along the x direction in the xz coordinate space. The control unit 24 can recognize the cluster 180 of point groups arranged along the x direction in the xz coordinate space as an object. Figure 50 In the cluster 180 of the point group Figure 49 The back side of the body of another vehicle 42A shown corresponds.

[0267] When an object is a parallel object, the point group is arranged along the z direction in the xz coordinate space. The control unit 24 can recognize that the object is a parallel object when a cluster 181 of point groups arranged along the z direction exists in the xz coordinate space. As mentioned above, examples of parallel objects include structures at the road end such as guardrails and soundproof walls on highways, or Figure 49 The side of the other vehicle 42A shown, etc. The cluster 181 of point groups arranged in the z direction in the xz coordinate space corresponds to objects arranged in parallel to the moving direction of the moving body 30 or the surface of the moving body 30 parallel to the moving direction of the object. The control unit 24 can exclude the cluster 181 of point groups arranged in the z direction from the objects to be detected in the object detection process. Here, the parallax corresponding to the parallel objects detected in the above clustering process is excluded. Figure 45 The representative parallax d in the process of step S902 is shown as e The clustering process may not detect all parallaxes corresponding to parallel objects. In this case, parallaxes corresponding to parallel objects may exist in the xz coordinate space, such as point cluster 181. Even if parallaxes corresponding to parallel objects, such as point cluster 181, exist in the xz coordinate space, they can be excluded from the object detection process in step S109.

[0268] The control unit 24 can detect the width of the object based on the width of the cluster 180 of the point group recognized as the object arranged in the x direction. Figure 45 The representative disparity d obtained by the process shown e Therefore, the control unit 24 can identify the position, lateral width, and height of the recognized object in the xz coordinate space.

[0269] In the process of step S110, the control unit 24 can output the information of the position, lateral width and height of the object recognized by the process of step S109 to other devices in the mobile body 30 through the output unit 22. For example, the control unit 24 can output this information to a display device in the mobile body 30. Figure 51 As shown, the display device in the mobile body 30 displays a detection frame 182 surrounding the image corresponding to the other vehicle 42A on the image of the first camera 11 or the second camera 12 based on the information obtained from the object detection device 20. Figure 27 In FIG, the detection frame 182 indicates the position of the detected object and the range it occupies in the image.

[0270] As described above, the object detection device 20 of the present disclosure enables fast processing speed and high-precision object detection. In other words, the object detection device 20 and the physical detection method of the present disclosure can improve object detection performance. Furthermore, the object detection device 20 does not limit detection targets to specific types of objects. The object detection device 20 can detect all objects on the road surface. The control unit 24 of the object detection device 20 can perform the first, second, third, and fourth processes without using information from images other than the first parallax image captured by the stereo camera 10. Therefore, the object detection device 20 need not perform processing to identify additional objects from the captured images, in addition to processing the first and second parallax images. Consequently, the object detection device 20 of the present disclosure can reduce the processing load on the control unit 24 involved in object recognition. This does not preclude the possibility of combining this with image processing performed by the object detection device 20 of the present disclosure on images directly obtained from the first camera 11 or the second camera 12. The object detection device 20 can also be combined with image processing techniques such as template matching.

[0271] In the above description of the processing executed by the control unit 24, to facilitate understanding of the present invention, the processing includes determinations and operations using various images. The processing using these images does not necessarily include actual image rendering. Processing substantially identical to the processing using these images is executed through information processing within the control unit 24.

[0272] The embodiments of the present invention have been described based on the accompanying drawings and embodiments, but it should be noted that it is easy for those skilled in the art to make various deformations or modifications based on the present invention. Therefore, it should be noted that these deformations or modifications are included in the scope of the present invention. For example, the functions included in each component or each step can be reconfigured in a logically non-contradictory manner, and multiple components or steps can be combined into one or divided. The embodiments of the present invention have been described with the device as the center, but the embodiments of the present invention can also be implemented as a method including the steps performed by each component of the device. The embodiments of the present invention can be implemented as a method executed by a processor possessed by the device, a program, or a storage medium having a program recorded thereon. It should be understood that these contents are also included in the scope of the present invention.

[0273] In the present invention, the descriptions of "first" and "second" are identifiers used to distinguish the structures. The structures distinguished by the descriptions of "first" and "second" in the present invention can exchange the numbers in the structures. For example, the first lens can exchange the "first" and "second" as identifiers with the second lens. The exchange of identifiers is carried out simultaneously. The structures can also be distinguished after the identifiers are exchanged. The identifiers can be deleted. The structures with deleted identifiers are distinguished by figure marks. The descriptions of identifiers such as "first" and "second" in the present invention should not be used to interpret the order of the structures, nor can they be used as a basis for the existence of identifiers with smaller numbers.

[0274] In the present invention, the x-direction, y-direction, and z-direction are provided for ease of description and may be interchanged. The structure of the present invention is described using an orthogonal coordinate system with the x-direction, y-direction, and z-direction as the directions of the respective axes. The positional relationship of the various structures of the present invention is not limited to being orthogonal. The u-coordinate and v-coordinate representing the coordinates of the image are provided for ease of description and may be interchanged. The origin and direction of the u-coordinate and v-coordinate are not limited to the content of the present invention.

[0275] In the above embodiment, the first camera 11 and the second camera 12 of the stereo camera 10 are arranged in the x direction. The configuration of the first camera 11 and the second camera 12 is not limited thereto. The first camera 11 and the second camera 12 may be arranged in a direction perpendicular to the road surface (y direction) or in a direction inclined relative to the road surface 41A. The number of cameras constituting the stereo camera 10 is not limited to two. The stereo camera 10 may include three or more cameras. For example, a total of four cameras, namely, two cameras arranged in a horizontal direction relative to the road surface and two cameras arranged in a vertical direction, may be used to obtain distance information with higher accuracy.

[0276] In the above embodiment, the stereo camera 10 and the object detection device 20 are mounted on the mobile object 30. However, the stereo camera 10 and the object detection device 20 are not limited to being mounted on the mobile object 30. For example, the stereo camera 10 and the object detection device 20 may be roadside equipment installed at an intersection, etc., and arranged to capture images including the road surface. For example, the roadside equipment can provide information such as detecting a first vehicle approaching from one of the intersecting roads at the intersection and notifying a second vehicle approaching from the other road of the first vehicle's approach.

[0277] Captions

[0278] 1.1A: Object Detection System

[0279] 10: Stereo Camera

[0280] 11: First Camera

[0281] 12: Second camera

[0282] 20: Object detection device

[0283] 21: Get Department

[0284] 22: Output

[0285] 23: Memory

[0286] 24: Control Department

[0287] 25: Generator

[0288] 30: Mobile

[0289] 40: First parallax image

[0290] 41, 42, 43, 44, 71, 72, 73, 74, 75, 76, 81, 82, 101: Parallax images

[0291] 41A: Road surface

[0292] 42A: Vehicle

[0293] 42a, 83, 84, 85, 102, 103, 104, 105, 151, 152, 153, 154, 161, 162, 163: Parallax pixels

[0294] 45: Frame line

[0295] 51: Graphics

[0296] 52: First straight line

[0297] 53: Approximate starting point

[0298] 54: Candidate line

[0299] 55: Second straight line

[0300] 60, 70, 80, 100, 150, 160: Second parallax image

[0301] 61: Part of the area

[0302] 61-1, 61: Partial area

[0303] 62: Area

[0304] 90, 106, 120, 170: First image

[0305] 91, 92, 107, 121, 122, 123, 124, 125, 171, 172: images

[0306] 101a: Partial area

[0307] 107a: Lower image

[0308] 107b: Upper image

[0309] 110, 140: UD chart

[0310] 110VP, 120VP: Vanishing Point

[0311] 111, 112, 113, 114, 115, 180, 181: point groups

[0312] 111L, 115L: Straight line

[0313] 131, 132, 133, 134: points

[0314] 131L, 132L, 133L, 134L, 135L: Curve

[0315] 141, 142: First parallax point

[0316] 143: Second parallax point

[0317] 172a, 172b: Partial images

[0318] 182: Detection frame

Claims

1. An object detection device, wherein: A processor is provided, the processor being configured to execute the following processing: A straight line of a specified length is detected by applying a Hough transform to the coordinate points of a UD map, wherein each coordinate point on the UD map has two-dimensional coordinates consisting of a first direction corresponding to the horizontal direction of a captured image generated by a stereo camera capturing a road surface and a second direction corresponding to the magnitude of a parallax in the parallax obtained from the captured image, wherein the UD map is a map that associates object parallaxes in parallaxes that satisfy specified conditions with object coordinate points in the coordinate points, and the object parallax corresponding to the detected straight line of the specified length is detected as a parallax corresponding to an object parallel to the direction of travel of the stereo camera. The processor is configured to convert, in the Hough transform, each object coordinate point associated with the object parallax and a straight line within a predetermined range based on a vanishing point into a Hough space.

2. The object detection device according to claim 1, wherein The processor is configured to exclude, from application of the Hough transform, coordinate points of the UD map having substantially the same magnitude of parallax and whose coordinates in the first direction of the two-dimensional coordinates are continuous within a predetermined range.

3. The object detection device according to claim 2, wherein: The predetermined range is set based on the lateral width of the detection target of the object detection device.

4. The object detection device according to any one of claims 1 to 3, wherein: The predetermined length is set based on the length of a structure provided along the road surface.

5. The object detection device according to any one of claims 1 to 4, wherein: The processor is configured to perform the following processing: a first process of estimating a shape of a road surface in real space based on a first disparity map generated based on the captured image, wherein the first disparity map is a map associating disparities obtained from the captured image with two-dimensional coordinates consisting of a first direction and a second direction intersecting the first direction; a second process of removing, from the first disparity map, disparities corresponding to a range of heights below a predetermined height from the road surface in real space based on the estimated shape of the road surface, thereby generating a second disparity map; and The process of generating the UD map based on the second disparity map.

6. The object detection device according to claim 5, wherein: The processor is configured to, in the process for generating the UD map, dividing the disparity map into a plurality of partial regions along the first direction of the disparity map, and generating a distribution representing the frequency of disparity for each of the plurality of partial regions, The disparity whose frequency exceeds a predetermined threshold is extracted as the target disparity.

7. An object detection system, wherein: have: a stereo camera that captures multiple images that have parallax with each other; and An object detection device comprising at least one processor, The at least one processor is configured to perform the following processing: A straight line of a specified length is detected by applying a Hough transform to coordinate points of a UD map, wherein the UD map is a map in which object disparities are associated with coordinate points on two-dimensional coordinates consisting of a first direction and a direction corresponding to the magnitude of the disparity, wherein the first direction corresponds to the horizontal direction of a captured image generated by a stereo camera capturing a road surface, the object disparity is a disparity obtained from the captured image that satisfies a specified condition, and the object disparity corresponding to the detected straight line of the specified length is detected as a disparity corresponding to an object parallel to the direction of travel of the stereo camera. In the Hough transform, each coordinate point associated with the object parallax and a straight line passing through a predetermined range based on a vanishing point are converted into a Hough space.

8. A mobile object, wherein: Equipped with object detection system, The object detection system has: a stereo camera that captures multiple images that have parallax with each other; and An object detection device comprising at least one processor, The at least one processor is configured to perform the following processing: A straight line of a specified length is detected by applying a Hough transform to coordinate points of a UD map, wherein the UD map is a map in which object disparities are associated with coordinate points on two-dimensional coordinates consisting of a first direction and a direction corresponding to the magnitude of the disparity, wherein the first direction corresponds to the horizontal direction of a captured image generated by a stereo camera capturing a road surface, the object disparity is a disparity obtained from the captured image that satisfies a specified condition, and the object disparity corresponding to the detected straight line of the specified length is detected as a disparity corresponding to an object parallel to the direction of travel of the stereo camera. In the Hough transform, each coordinate point associated with the object parallax and a straight line passing through a predetermined range based on a vanishing point are converted into a Hough space.

9. A method for object detection, wherein: The following steps are involved: Detecting a straight line of a specified length by applying a Hough transform to coordinate points of a UD map, wherein the UD map is a map in which object disparities are associated with coordinate points on two-dimensional coordinates consisting of a first direction and a direction corresponding to the magnitude of the disparity, wherein the first direction corresponds to the horizontal direction of a captured image generated by a stereo camera capturing a road surface, and the object disparity is a disparity satisfying a specified condition among the disparities obtained from the captured image; and The object disparity corresponding to the detected straight line of the predetermined length is detected as the disparity corresponding to the object parallel to the moving direction of the stereo camera, The detecting of the parallax corresponding to the parallel object includes converting, in the Hough transform, a straight line passing through each coordinate point associated with the object parallax and a predetermined range based on a vanishing point into a Hough space.

Images