Road surface detection device, object detection device, object detection system, mobile object, and object detection method

Abstract

The object detection device includes a processor. The processor acquires or generates a first parallax image based on the output of a stereo camera mounted on a moving object. The first parallax image is formed by arranging pixels representing parallax on a two-dimensional plane defined by a first direction corresponding to the baseline longitudinal direction of the stereo camera and a second direction intersecting the first direction. The processor approximates the relationship between the coordinates of the road surface in the second direction in the moving object's direction of travel, as included in the first parallax image, and the parallax representing the road surface using two straight lines.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Japanese Patent Application No. 2019-166595, filed on September 12, 2019, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to a road surface detection device, an object detection device, an object detection system, a mobile object, and an object detection method. Background Art

[0004] In recent years, object detection devices using stereo cameras have been installed in mobile vehicles such as automobiles for object detection and distance measurement. These object detection devices calculate the distance distribution (or parallax distribution) across multiple images captured by multiple cameras, and use this information to identify obstacles (for example, see Patent Document 1).

[0005] Prior art literature

[0006] Patent Literature

[0007] Japanese Patent Application Laid-Open No. 5-265547 Summary of the Invention

[0008] The road surface detection device disclosed herein includes a processor. The processor acquires or generates a first disparity map based on the output of a stereo camera that captures an image including a road surface. The first disparity map is a map that associates disparities obtained from the stereo camera output with two-dimensional coordinates consisting of a first direction and a second direction intersecting the first direction, wherein the first direction corresponds to the horizontal direction of the image captured by the stereo camera. The processor approximates the relationship between the coordinates of the road surface in the second direction, included in the first disparity map, and the disparity representing the road surface using two straight lines.

[0009] The object detection device disclosed herein includes a processor. The processor acquires or generates a first disparity map, the first disparity map being generated based on the output of a stereo camera that captures an image including a road surface. The first disparity map is a map that associates disparities obtained from the output of the stereo camera with two-dimensional coordinates consisting of a first direction and a second direction intersecting the first direction, wherein the first direction corresponds to the horizontal direction of the image captured by the stereo camera. The processor approximates the relationship between the coordinates of the road surface in the second direction, included in the first disparity map, and the disparity representing the road surface using two straight lines. The processor is configured to detect objects using the disparity representing the road surface approximated by the straight lines.

[0010] The object detection system disclosed herein comprises: a stereo camera that captures multiple images having parallaxes; and an object detection device including at least one processor. The processor acquires or generates a first disparity map, which is generated based on the output of the stereo camera that captures an image including a road surface. The first disparity map is a map that associates disparities obtained from the output of the stereo camera with two-dimensional coordinates consisting of a first direction and a second direction intersecting the first direction, wherein the first direction corresponds to the horizontal direction of the image captured by the stereo camera. The processor approximates the relationship between the coordinates of the road surface in the second direction, included in the first disparity map, and the disparity representing the road surface using two straight lines. The processor is configured to detect objects using the disparity representing the road surface approximated by the straight lines.

[0011] The mobile object disclosed herein is equipped with an object detection system. The object detection system includes a stereo camera that captures multiple images having parallaxes with each other, and an object detection device having at least one processor. The processor acquires or generates a first parallax map, which is generated based on the output of a stereo camera that captures an image including a road surface. The first parallax map is a map that associates parallaxes obtained from the output of the stereo camera with two-dimensional coordinates consisting of a first direction and a second direction intersecting the first direction, wherein the first direction corresponds to the horizontal direction of the image captured by the stereo camera. The processor approximates the relationship between the coordinates of the road surface in the second direction included in the first parallax map and the parallax representing the road surface using two straight lines. The processor is configured to detect an object using the parallax representing the road surface approximated by the straight lines.

[0012] The object detection method disclosed herein includes acquiring or generating a first disparity map, the first disparity map being generated based on the output of a stereo camera that captures an image containing a road surface. The first disparity map is a map that associates disparities obtained from the stereo camera output with two-dimensional coordinates consisting of a first direction and a second direction intersecting the first direction, wherein the first direction corresponds to the horizontal direction of the image captured by the stereo camera. The object detection method approximates the relationship between the coordinates of the road surface in the second direction, included in the first disparity map, and the disparity representing the road surface using two straight lines. The object detection method includes detecting objects using the disparity representing the road surface approximated by the straight lines. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a block diagram showing a schematic configuration of an object detection system according to one embodiment of the present disclosure.

[0014] Figure 2 is a schematic diagram showing the installation Figure 1 Side view of a moving body of an object detection system.

[0015] Figure 3 is a schematic diagram showing the installation Figure 1 Front view of a moving body of an object detection system.

[0016] Figure 4 This is a block diagram showing a schematic structure of another object detection system according to the present disclosure.

[0017] Figure 5 It shows Figure 1 A flowchart of an example of processing performed by an object detection device.

[0018] Figure 6 A diagram illustrating an example of a first parallax image acquired or generated by the object detection device.

[0019] Figure 7 This is a flowchart showing an example of processing for estimating the road surface shape.

[0020] Figure 8 This is a flowchart showing an example of a process of extracting a road surface candidate parallax from a first parallax image.

[0021] Figure 9 A diagram illustrating the positional relationship between a road surface and a stereo camera.

[0022] Figure 10 3 is a diagram illustrating the procedure for extracting road surface candidate parallaxes.

[0023] Figure 11 FIG. 1 is a diagram showing a range on a road surface obtained by converting a parallax histogram.

[0024] Figure 12 is the road parallax d r A dv correlation diagram showing an example of the relationship between the vertical coordinate (v coordinate).

[0025] Figure 13 This is a diagram for explaining a method of detecting whether an object other than road surface parallax is included.

[0026] Figure 14 is the road parallax d r Flowchart for processing the relationship between the vertical left side of the image (v coordinate) using a straight line approximation.

[0027] Figure 15 This is an illustration of the road parallax d generated by the first straight line. r An approximate diagram of .

[0028] Figure 16 This is a diagram explaining the method of determining the second straight line.

[0029] Figure 17 is a graph showing the road parallax d r A diagram showing an example of the result of linearly approximating the relationship between the vertical coordinate (v coordinate) of the image.

[0030] Figure 18 A diagram illustrating an example of a second parallax image from which unnecessary parallax is removed.

[0031] Figure 19 A diagram showing an example of a second parallax image.

[0032] Figure 20 is a diagram showing one of the areas on the second parallax image where a histogram is created.

[0033] Figure 21 This is a diagram explaining a method of determining object disparity using a histogram.

[0034] Figure 22 A diagram illustrating a method of acquiring height information of object parallax.

[0035] Figure 23 A diagram illustrating a method of acquiring height information of object parallax.

[0036] Figure 24 This is a diagram showing an example of the distribution of a point cloud representing object parallax in the ud space.

[0037] Figure 25 This is a diagram of the road surface viewed from the height direction (y direction).

[0038] Figure 26 It is a graph that converts object parallax into a point cloud on the xz plane in real space.

[0039] Figure 27 This is a diagram showing an example of a method for outputting object detection results. DETAILED DESCRIPTION

[0040] Due to various factors such as the shape of the road and the presence of structures and other objects on the road, the object detection device may not achieve high detection performance. The road surface detection device, object detection device, object detection system, object detection method and mobile object disclosed in the present invention can improve the performance of object detection.

[0041] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be noted that the figures used in the following description are schematic. The dimensions, proportions, etc. in the drawings do not necessarily correspond to the actual ones. The figures showing images taken by a camera and parallax images, etc. include figures made for illustration. These images are different from the images actually taken or processed. In addition, in the following description, the "subject" is an object photographed by a camera. The "subject" includes objects, roads, the sky, etc. An "object" has a specific position and size in space. An "object" may also be referred to as a "three-dimensional object."

[0042] like Figure 1 As shown, the object detection system 1 includes a stereo camera 10 and an object detection device 20. The stereo camera 10 and the object detection device 20 can communicate via wired or wireless communication. The stereo camera 10 and the object detection device 20 can communicate via a network. The network can include, for example, a wired or wireless LAN (Local Area Network), a CAN (Controller Area Network), etc. The stereo camera 10 and the object detection device 20 can be housed in the same housing and constructed integrally. The stereo camera 10 and the object detection device 20 can be located in a mobile body 30 described later and can communicate with an ECU (Electronic Control Unit) in the mobile body 30.

[0043] A "stereo camera" is a plurality of cameras that have parallax with each other and cooperate with each other. A stereo camera includes at least two or more cameras. By making the multiple cameras cooperate, a stereo camera can shoot an object from multiple directions. A stereo camera can be a device that includes multiple cameras in one housing. A stereo camera can be a device that includes two or more cameras that are independent of each other and located at positions separated from each other. A stereo camera is not limited to multiple cameras that are independent of each other. In the present disclosure, for example, a camera having an optical mechanism that guides light incident on two separate points to one light receiving element can be used as a stereo camera. In the present disclosure, multiple images of the same subject shot from different viewpoints are referred to as "stereo images."

[0044] The stereo camera 10 includes a first camera 11 and a second camera 12. The first camera 11 and the second camera 12 each include an optical system defining an optical axis OX and an imaging device. The first camera 11 and the second camera 12 each have a different optical axis OX. In this specification, only one reference numeral OX is used to represent the optical axis OX of both the first camera 11 and the second camera 12. The imaging device includes a CCD image sensor (Charge-Coupled Device Image Sensor) and a CMOS image sensor (Complementary MOS Image Sensor). The imaging devices included in the first camera 11 and the second camera 12 can be located in the same plane perpendicular to the optical axis OX of each camera. The first camera 11 and the second camera 12 generate image signals representing images formed by the imaging devices. In addition, the first camera 11 and the second camera 12 can perform any processing on the captured images, such as distortion correction, brightness adjustment, contrast adjustment, and gamma correction.

[0045] The optical axes OX of the first and second cameras 11, 12 are oriented in a direction that allows them to capture the same subject. The optical axes OX and positions of the first and second cameras 11, 12 are determined so that the captured images include at least the same subject. In one embodiment, the optical axes OX of the first and second cameras 11, 12 are oriented parallel to each other. This parallelism is not strictly parallel; it allows for assembly and installation variations, as well as variations caused by time. In another embodiment, the optical axes OX of the first and second cameras 11, 12 are not necessarily parallel and can face different directions. Even if the optical axes OX of the first and second cameras 11, 12 are not parallel, stereoscopic images can be generated by converting the images within the stereo camera 10 or the object detection device 20. The distance between the optical centers of the first and second cameras 11, 12 of the stereo camera 10 is referred to as the baseline length. The baseline length is equivalent to the distance between the lens centers of the first and second cameras 11, 12. The direction connecting the optical centers of the first and second cameras 11, 12 of the stereo camera 10 is referred to as the baseline length direction.

[0046] The first camera 11 and the second camera 12 are separated from each other in a direction intersecting the optical axis OX. In one embodiment of the multiple embodiments, the first camera 11 and the second camera 12 are arranged in the left-right direction. When facing forward, the first camera 11 is located on the left side of the second camera 12. When facing forward, the second camera 12 is located on the right side of the first camera 11. Since the positions of the first camera 11 and the second camera 12 are different, the positions of the subjects corresponding to each other in the two images captured by each camera are different. The first image output from the first camera 11 and the second image output from the second camera 12 are stereo images captured from different viewpoints. The first camera 11 and the second camera 12 capture the subject at a predetermined frame rate (for example, 30fps).

[0047] like Figure 2 and Figure 3 As shown, Figure 1 The object detection system 1 is mounted on a moving object 30. Figure 2 As shown in the side view of FIG, the first camera 11 and the second camera 12 are arranged to capture images in front of the mobile object 30. The first camera 11 and the second camera 12 are arranged to capture images that include the road surface. In other words, the stereo camera 10 is arranged to capture images that include the road surface. In one embodiment, the optical axis OX of the optical system of each of the first camera 11 and the second camera 12 is arranged to be substantially parallel to the front of the mobile object 30.

[0048] In this application, the direction in which the moving body 30 is traveling forward is referred to as the front or z-direction. The direction opposite to the front is referred to as the rear. The left and right directions are defined based on the state facing the front of the moving body 30. The direction perpendicular to the z-direction and from left to right is referred to as the x-direction. The x-direction may coincide with the baseline length direction. The direction perpendicular to and upward from the road surface near the moving body 30 is referred to as the height direction or the y-direction. The y-direction may be perpendicular to the x-direction and the z-direction. The x-direction is also referred to as the horizontal direction. The y-direction is also referred to as the vertical direction. The z-direction is also referred to as the depth direction.

[0049] "Mobile objects" in this disclosure may include, for example, vehicles and aircraft. Vehicles may include, for example, automobiles, industrial vehicles, rail vehicles, household vehicles, and fixed-wing aircraft taxiing on runways. Automobiles may include, for example, cars, trucks, buses, motorcycles, and trolleybuses. Industrial vehicles may include, for example, those used in agriculture and construction. Industrial vehicles may include, for example, forklifts and golf carts. Agricultural industrial vehicles may include, for example, tractors, tillers, rice transplanters, harvesters, combines, and lawn mowers. Construction industrial vehicles may include, for example, bulldozers, scrapers, excavators, truck cranes, dump trucks, and road rollers. Vehicles may include human-powered vehicles. The classification of vehicles is not limited to the above examples. For example, automobiles may include industrial vehicles that can travel on roads. The same vehicle may be included in multiple classifications. Aircraft may include, for example, fixed-wing aircraft and rotary-wing aircraft.

[0050] The vehicle 30 of the present disclosure travels on a travel path including a road, a track, etc. The surface of the travel path on which the vehicle 30 travels is called a road surface.

[0051] The first camera 11 and the second camera 12 are mounted at different locations on the mobile object 30. In one embodiment, the first camera 11 and the second camera 12 are mounted inside the mobile object 30, which is a vehicle, and can capture images of the exterior of the mobile object 30 through the windshield. For example, the first camera 11 and the second camera 12 are positioned in front of the rearview mirror or on the dashboard. In one embodiment, the first camera 11 and the second camera 12 can be attached to any of the vehicle's front bumper, fender grille, side fenders, headlight module, hood, and the like.

[0052] Object detection device 20 includes an acquisition unit 21, an image processing unit 22 (processor), a memory 23, and an output unit 24. Object detection device 20 can be placed at any position in moving object 30. For example, object detection device 20 can be placed in the instrument panel of moving object 30.

[0053] The acquisition unit 21 is an input interface of the object detection device 20 that receives information input from the stereo camera 10 and other devices. The acquisition unit 21 can use physical connectors and wireless communication devices. Physical connectors include electrical connectors corresponding to the transmission of electrical signals, optical connectors corresponding to the transmission of optical signals, and electromagnetic connectors corresponding to the transmission of electromagnetic waves. Electrical connectors include connectors that comply with IEC60603, connectors that comply with USB standards, connectors corresponding to RCA terminals, connectors corresponding to S terminals specified in EIAJCP-1211A, connectors corresponding to D terminals specified in EIAJ RC-5237, connectors that comply with HDMI (registered trademark) standards, and connectors corresponding to coaxial cables including BNCs. Optical connectors include various connectors that comply with IEC61754. Wireless communication devices include wireless communication devices that comply with various standards such as Bluetooth (registered trademark) and IEEE802.11. Wireless communication devices include at least one antenna.

[0054] The acquisition unit 21 can receive image data of images captured by the first camera 11 and the second camera 12, respectively. The image data received by the acquisition unit 21 is transmitted to the image processing unit 22. The acquisition unit 21 can be compatible with the transmission method of the imaging signal of the stereo camera 10. The acquisition unit 21 can be connected to the output interface of the stereo camera 10 via a network.

[0055] The image processing unit 22 includes one or more processors. Processors include general-purpose processors that load specific programs and execute specific functions, and special-purpose processors that specialize in specific processing. Special-purpose processors include application-specific integrated circuits (ASICs). Processors include programmable logic devices (PLDs). PLDs include FPGAs (Field-Programmable Gate Arrays). The image processing unit 22 can be any of an SoC (System-on-a-Chip) and a SiP (System In a Package) in which one or more processors cooperate. The processing performed by the image processing unit 22 can also be regarded as processing performed by a processor.

[0056] The image processing unit 22 includes functional blocks: a parallax image generation unit 25, a road surface detection unit 26 (road surface detection device), an unnecessary parallax removal unit 27, a clustering unit 28, and a grouping unit 29. The parallax image generation unit 25 generates a first parallax image based on the first and second images output from the stereo camera 10. The first parallax image is an image in which pixels representing parallax are arranged on a two-dimensional plane formed by a horizontal direction corresponding to the horizontal direction of the image captured by the stereo camera 10 and a vertical direction intersecting the horizontal direction. The horizontal direction is the first direction. The vertical direction is the second direction. The horizontal and vertical directions may be orthogonal to each other. The horizontal direction corresponds to the width direction of the road surface. The horizontal direction is a direction parallel to the horizontal line when the image captured by the stereo camera 10 contains a horizontal line. The vertical direction may correspond to the direction of gravity in real space. The road surface detection unit 26, the unnecessary parallax removal unit 27, the clustering unit 28, and the grouping unit 29 perform a series of processes to detect objects based on the first parallax image.

[0057] In the information processing in the image processing unit 22, the first parallax image undergoes various operations as a first parallax map. In the first parallax map, parallax information obtained from the output of the stereo camera 10 is associated with two-dimensional coordinates consisting of horizontal and vertical directions. These operations include calculations and writing to and reading from the memory 23. The first parallax image can also be considered a first parallax map. In the following description, processing of the first parallax image can also be considered processing of the first parallax map.

[0058] Each functional block of the image processing unit 22 can be a hardware module or a software module. The processing performed by each functional block can also be considered as processing performed by the image processing unit 22. The image processing unit 22 can perform all actions of each functional block. The processing that the image processing unit 22 causes any functional block to perform can be performed by the image processing unit 22 itself. The road surface detection unit 26 can also be referred to as a road surface detection device. The road surface detection device can be composed of hardware shared with other functional blocks or hardware separate from other functional blocks. In addition to the road surface detection unit 26, the road surface detection device may also include a parallax image generation unit 25.

[0059] The memory 23 stores programs used for various processes and information used in operations. The memory 23 includes a volatile memory and a non-volatile memory. The memory 23 includes a memory independent of the processor and a memory built into the processor.

[0060] The output unit 24 is the output interface of the object detection device 20. It can output the processing results of the object detection device 20 to other devices within the mobile object 30 or to devices external to the mobile object 30, such as other vehicles and roadside equipment. Other devices that can appropriately use the information received from the object detection device 20 include driving assistance systems such as automatic cruise control and safety equipment such as automatic braking systems. Similar to the acquisition unit 21, the output unit 24 includes various interfaces for both wired and wireless communication. For example, the output unit 24 includes a CAN interface to communicate with other devices within the mobile object 30.

[0061] The object detection device 20 can be configured to implement the processing performed by the image processing unit 22 described below by reading a program recorded in a non-temporary computer-readable medium. Non-temporary computer-readable media include magnetic storage media, optical storage media, magneto-optical storage media, and semiconductor storage media, but are not limited to these. Magnetic storage media include magnetic disks, hard disks, and magnetic tapes. Optical storage media include CDs (Compact Discs), DVDs, Blu-ray Discs (Blu-ray (registered trademark) Discs), and other optical disks. Semiconductor storage media include ROMs (Read Only Memory), EEPROMs (Electrically Erasable Programmable Read-Only Memory), and flash memories.

[0062] like Figure 4 As shown, in an object detection system 1A according to another embodiment of the present disclosure, the parallax image generation unit 25 may be mounted on hardware independent of the object detection device 20. Figure 4 In, with Figure 1 The same or similar components are used Figure 1 The same symbols are used to represent them. Figure 4 The parallax image generating unit 25 in FIG. 2 can also be regarded as a parallax image generating device. Figure 4 The parallax image generating unit 25 includes a processor. Figure 4 The processor included in the parallax image generating section 25 generates a first parallax image based on the first image and the second image outputted from the first camera 11 and the second camera 12 of the stereo camera 10 , respectively. Figure 4 The acquisition unit 21 included in the object detection device 20 acquires the first parallax image from the parallax image generation unit 25 . Figure 4 The object detection device 20 and the parallax image generation unit 25 can be regarded as one object detection device 20. Figure 4 In the object detection device 20 thus configured, the functions of the road surface detection unit 26, the unnecessary parallax removal unit 27, the clustering unit 28, and the grouping unit 29 of the image processing unit 22 are similar to those of the Figure 1 The corresponding functional blocks are the same.

[0063] The following will refer to Figure 5 The flowchart further illustrates the processing performed by each functional block of the image processing unit 22. Figure 5 1 is a flowchart for explaining the overall processing of the object detection method executed by the object detection device 20 .

[0064] First, in detail Figure 5 Before describing the processing performed in each step of the flowchart, the outline and purpose of the processing in each step will be briefly described.

[0065] Step S101 is a step of acquiring or generating a first parallax image as a target for object detection at a stage prior to the first process described later. Step S101 is performed by the acquiring unit 21 or the parallax image generating unit 25 .

[0066] Step S102 is the step of estimating the shape of the road surface. The processing performed in step S102 is also referred to as the first processing. Step S102 is performed by the road surface detection unit 26. By estimating the road surface shape, the parallax representing the road surface relative to the vertical coordinate can be found in the first parallax image. The road surface shape is necessary for removing unnecessary parallax in subsequent processing and / or determining the height position of the road surface in real space. The processing in step S102 helps improve the performance of object detection in the following steps S103 to S106.

[0067] Step S103 is a step of generating a second parallax image from which unnecessary parallax is removed. The processing performed in step S103 may also be referred to as a second processing. The second processing is performed by the unnecessary parallax removal unit 27. Unnecessary parallax is the parallax represented by pixels corresponding to a subject whose height from the road surface in real space is included in a predetermined range. For example, unnecessary parallax includes the parallax of the road surface included in the first parallax image and the parallax of a structure included in the air portion. By removing unnecessary parallax from the first parallax image, the possibility of erroneously detecting white lines on the road surface and structures existing above the road surface as objects on the road surface is reduced. As a result, the accuracy of object detection is improved.

[0068] In the information processing performed by the image processing unit 22, the second parallax image is subjected to various operations as a second parallax image obtained by removing unnecessary parallax information from the parallax information included in the first parallax image. The second parallax image can also be considered the second parallax map. In the following description, processing of the second parallax image can also be considered processing of the second parallax map.

[0069] Step S104 is a step of determining object parallax at each horizontal coordinate for the second parallax image. Step S104 is performed by the clustering unit 28. Object parallax is parallax of an area determined to be recognizable as an object in real space under predetermined conditions.

[0070] Step S105 is a step for calculating height information related to the object parallax based on the object parallax distribution in the second parallax image and the road surface shape estimated in step S102. Step S105 is performed by clustering unit 28. The processing performed in steps S104 and S105 can also be referred to as the third processing. If height information is not required, step S105 can be omitted.

[0071] Step S106 is a step of detecting an object by transforming the object parallax information into real space coordinates and extracting a set (group) of object parallaxes. The process performed in step S106 may also be referred to as a fourth process. The fourth process is performed by the grouping unit 29.

[0072] Step S107 is the step of outputting the detected object information from the output unit 24. Based on the results of step S106, the detected object position and width information at the angle of the stereo camera 10 can be obtained. The information obtained in step S105 includes the height information of the detected object. This information can be provided to other devices within the mobile object 30.

[0073] Next, the details of each step will be described.

[0074] First, the image processing unit 22 acquires or generates a first parallax image (step S101). Figure 1 In the object detection system 1 shown in FIG. 1 , the image processing unit 22 generates a first parallax image based on the first image and the second image acquired by the acquisition unit 21 . The generation of the first parallax image is performed by the parallax image generation unit 25 . Figure 4 In the object detection system 1A shown, the image processing unit 22 acquires the first parallax image generated by the parallax image generating unit 25 via the acquiring unit 21. The image processing unit 22 may store the first parallax image in the memory 23 for subsequent processing.

[0075] The parallax image generating unit 25 generates a first parallax image by calculating the distribution of parallax between the first image acquired from the first camera 11 and the second image acquired from the second camera 12. The method of generating the first parallax image is well known and will be briefly described below.

[0076] The parallax image generating unit 25 divides the image of one of the first image and the second image (for example, the first image) into a plurality of small areas. The small area may be a rectangular area in which a plurality of pixels are arranged in the vertical and horizontal directions. For example, the small area may be composed of three pixels in the vertical direction and three pixels in the horizontal direction. The number of pixels contained in the vertical and horizontal directions of the small area is not limited to three. The number of pixels contained in the vertical and horizontal directions of the small area may be different. The parallax image generating unit 25 transforms each image divided into a plurality of small areas and the image of the other party (for example, the second image) in the horizontal direction and matches them. For image matching, a method using the SAD (Sum of Absolute Difference) function is known. This represents the sum of the absolute values of the differences between the brightness values within the small areas. When the SAD function is minimum, the two images are judged to be most similar. The matching of stereo images is not limited to the method using the SAD function, and other methods may also be used.

[0077] The parallax image generation unit 25 calculates the parallax of each small area based on the difference between the horizontal pixel positions of the two areas when matching the first image and the second image. The size of the parallax can be expressed in units of the horizontal width of the pixel. By performing interpolation processing, the size of the parallax can be calculated with an accuracy of less than one pixel. The size of the parallax corresponds to the distance between the subject photographed by the stereo camera 10 and the stereo camera 10 in real space. If the parallax is large, it means the distance is close; if the parallax is small, it means the distance is far. The parallax image generation unit 25 generates a first parallax image representing the calculated parallax distribution. Hereinafter, the pixels representing the parallax constituting the first parallax image will be referred to as parallax pixels. The parallax image generation unit 25 can generate a parallax image with the same clarity as the pixels of the original first and second images.

[0078] Figure 6 is a diagram illustrating the first parallax image. Figure 6 In FIG, another vehicle 42 is traveling on a road surface 41 in front of the moving object 30 .

[0079] like Figure 6 As shown, in the first parallax image, the pixels representing the parallax are located on a two-dimensional plane formed by the horizontal direction (first direction) and the vertical direction (second direction) orthogonal to the horizontal direction of the stereo camera 10. The coordinate representing the position in the horizontal direction is called the u coordinate. The coordinate representing the position in the vertical direction is called the v coordinate. The u coordinate and the v coordinate are called image coordinates. In each figure represented by the image of the present disclosure, the u coordinate is the coordinate from left to right. The v coordinate is the coordinate from top to bottom. The origin of the uv coordinate space can be the upper left end of the first parallax image. The u coordinate and the v coordinate can be expressed in pixels.

[0080] The parallax image generating unit 25 can replace the difference in display parallax with the difference in brightness or color of the pixels. Figure 6 In , different shading is used to represent parallax for illustration purposes. Figure 6 In the case of , the darker the shadow, the smaller the parallax, and the lighter the shadow, the larger the parallax. Figure 6 Areas with equal shading in the image are within the specified parallax range. In actual first-parallax images, there are areas where parallax is easily acquired and areas where it is difficult to acquire parallax in the UV coordinate space. For example, parallax is difficult to acquire in spatially uniform areas of a subject, such as a car window, or in areas overexposed by sunlight reflection. In first-parallax images, if objects or structures are present, they appear with a different brightness or color than the more distant background.

[0081] The parallax image generating unit 25 does not need to display the first parallax image as an image after calculating the parallax. The parallax image generating unit 25 may generate and store information of the first parallax image as a first parallax map within the image processing unit 22 and perform necessary processing.

[0082] After step S101, the image processing unit 22 performs a first process of estimating the shape of the road surface 41 based on the first parallax image (step S102). The first process is performed by the road surface detection unit 26. Figure 7 、 8 14 and 15, the road shape estimation process performed by the road detection unit 26 is described. First, the road detection unit 26 extracts a road disparity candidate d from the first disparity image. c (Step S201). Road surface candidate disparity d c is the road disparity d collected from the first disparity image r The parallax with a high probability of corresponding. Road parallax d r It refers to the parallax of the road surface 41 area. r is the parallax excluding objects on the road surface 41. Road surface parallax d r Indicates the distance to the corresponding position on the road. Road parallax d r are collected as having close values at positions with the same v coordinate

[0083] Road surface candidate disparity d c The details of the extraction process are in Figure 8 As shown in the flow chart. Figure 8As shown, the road surface detection unit 26 calculates an initial value d0 of the road surface candidate disparity based on the installation position of the stereo camera 10, that is, an initial value of the disparity used to calculate the road surface candidate disparity (step S301). The initial value d0 of the road surface candidate disparity is the initial value of the road surface candidate disparity at the road surface candidate disparity extraction position closest to the stereo camera 10. The road surface candidate disparity extraction position closest to the stereo camera 10 can be set, for example, within a range of 1 to 10 meters from the stereo camera 10.

[0084] like Figure 9 As shown, in the stereo camera 10, the road surface height Y is the height of the stereo camera 10 from the photographed road surface 41 in the vertical direction. In addition, the road surface height Y0 is the height of the installation position of the stereo camera 10 from the road surface 41. Due to the undulations of the road, the road surface height Y may vary depending on the distance from the stereo camera 10. Therefore, the road surface height Y at a position far from the stereo camera 10 is inconsistent with the road surface height Y0 at the installation position of the stereo camera 10. In one embodiment of the multiple embodiments, the optical axes OX of the first camera 11 and the second camera 12 of the stereo camera 10 are arranged parallel to each other and face forward. Figure 9 In this case, Z represents the horizontal distance to a specific road surface position. Let the baseline length of the stereo camera 10 be B, and the longitudinal image size be TOTALv. In this case, the road surface parallax d of the road surface 41 captured at any longitudinal coordinate (v coordinate) is s The relationship with the road surface height Y is given by the following mathematical formula and is independent of the horizontal distance Z.

[0085] d s =B / Y×(v-TOTALv / 2) (1)

[0086] The road parallax d calculated by equation (1) s It is also called "geometrically estimated road parallax". s Indicates the geometrically estimated road surface parallax.

[0087] The road surface candidate disparity initial value d0 is obtained by assuming that the road surface 41 is flat and parallel to the stereo camera 10 and the road surface candidate disparity d is closest to the stereo camera 10. cThe coordinates d0 of the road candidate parallax are calculated based on the optical axis OX between the extraction positions of the first camera 11 and the second camera 12. In this case, the v coordinate at the extraction position of the road candidate parallax closest to the position of the stereo camera 10 on the first parallax image is set to a specific coordinate v0. The coordinate v0 is the initial value of the v coordinate when the road candidate parallax is extracted. The coordinate v0 is between TOTALv / 2 and TOTALv. The coordinate v0 is located at the bottom (the larger side of the v coordinate) within the range of image coordinates capable of calculating the parallax. The coordinate v0 can be TOTALv corresponding to the bottom row in the first parallax image. The road candidate parallax initial value d0 can be determined by substituting v0 and Y0 into v and Y in mathematical formula (1).

[0088] The road surface detection unit 26 calculates the disparity collection threshold for the first row with a vertical v-coordinate of v0 based on the road surface candidate disparity initial value d0 (step S302). A row is an array of horizontally arranged pixels with the same v-coordinate on the first disparity image. The disparity collection threshold includes an upper threshold, which serves as the upper limit for collecting disparities, and a lower threshold, which serves as the lower limit for collecting disparities. The disparity collection threshold is set above and below the road surface candidate disparity initial value d0 based on a predetermined rule to include the road surface candidate disparity initial value d0. Specifically, the road surface disparity is defined as the upper and lower thresholds of the disparity collection threshold when the road surface height Y changes by a predetermined road surface height change ΔY from the state where the road surface candidate disparity initial value d0 is calculated. That is, the lower threshold of the disparity collection threshold is obtained by subtracting the disparity corresponding to the road surface height change ΔY from the road surface candidate disparity initial value d0. The upper threshold of the disparity collection threshold is obtained by adding the disparity corresponding to the road surface height change ΔY to the road surface candidate disparity initial value d0. The specific lower and upper thresholds of the disparity collection threshold can be obtained by changing the value of Y in the mathematical formula (1).

[0089] Next, the road surface detection unit 26 repeats the processing between step S303 and step S307. First, the road surface detection unit 26 processes the line at the bottom of the first parallax image where the v coordinate is v0 (step S303).

[0090] The road surface detection unit 26 collects disparities using the disparity collection threshold (step S304). The road surface detection unit 26 collects disparity pixels having a disparity between the lower threshold and the upper threshold of the disparity collection threshold as candidate road surface disparities d for each disparity pixel arranged in the horizontal direction when the v coordinate is v0 in the first disparity image. cThat is, the road surface detection unit 26 determines that the parallax pixels having the parallax within the predetermined limit range based on the road surface candidate parallax initial value d0 calculated using the mathematical formula (1) are candidates for the parallax pixels representing the correct parallax of the road surface 41. The road surface detection unit 26 determines that the parallax pixels corresponding to the parallax pixels determined as candidates for the parallax pixels are candidates for the road surface parallax d c , where the candidate pixel determined to be a parallax pixel represents the correct parallax of the road surface 41. In this way, the road surface detection unit 26 can reduce the possibility of erroneously determining parallax outside of the road surface 41, such as objects or structures on the road surface 41, as parallax of the road surface 41. This improves the accuracy of detecting the road surface 41.

[0091] When all the disparity pixels when the v coordinate is v0 are determined in step S304, the road surface detection unit 26 calculates the collected road surface candidate disparity d c Take the average to calculate the average road candidate disparity d av , that is, the road surface candidate disparity d c The road surface detection unit 26 can calculate the average value of each road surface candidate disparity d c and its uv coordinates and the average road candidate disparity d when the v coordinate is v0 av Stored in memory 23.

[0092] After step S305, the road surface detection unit 26 calculates the average road surface candidate disparity d when the v coordinate is v0 calculated in step S305. av , the parallax collection threshold is calculated for each parallax pixel of the row above, that is, the row with v coordinate v=v0-1 (step S306). The road surface detection unit 26 changes the road surface height Y so that the average road surface candidate parallax d calculated in step S305 when the v coordinate is v0 is av The road surface detection unit 26 calculates the geometrically estimated road surface parallax d when the v coordinate is v0-1 by substituting v0-1 into the road surface height Y instead of v0 in the road surface detection unit 26. s Similar to step S302, the road surface detection unit 26 can estimate the road surface parallax d from the geometry. s The parallax obtained by subtracting the parallax corresponding to the predetermined road surface height change ΔY is set as the lower limit threshold of the parallax collection threshold. The road surface detection unit 26 can use the geometrically estimated road surface parallax d s The parallax obtained by adding the parallax corresponding to the predetermined road surface height change amount ΔY is set as the upper limit threshold of the parallax collection threshold.

[0093] After step S306, the road surface detection unit 26 determines the geometrically estimated road surface parallax d calculated by equation (1). sIs it greater than a predetermined value? The predetermined value is, for example, one pixel. The road surface detection unit 26 geometrically estimates the road surface parallax d s If it is greater than 1, the process returns to step S303 (step S307). In step S303, the road surface detection unit 26 calculates the road surface candidate parallax d c The extracted object moves to a row above one pixel. That is, the road surface candidate disparity d c When the extraction target is in the row with v coordinate v0, the road surface detection unit 26 changes the v coordinate of the row of the road surface detection target to v0-1. Figure 10 As shown, the road candidate disparity d c When the calculation target of is the nth row, the road surface detection unit 26 changes the row of the road surface detection target to the n+1th row. Figure 10 For ease of illustration, the vertical width of each row is exaggerated. In reality, each row is one pixel high. In this case, the v coordinate of row (n+1) is 1 less than the v coordinate of row (n).

[0094] Each process of steps S304 to S306 regarding the n+1th row is the same as the process performed on the row when the v coordinate is v0. In step S304, the road surface detection unit 26 collects the road surface candidate disparity d using the disparity collection threshold value calculated for the nth row in step S306. c In step S305, the road surface detection unit 26 collects the road surface candidate disparity d c Take the average to calculate the average road candidate disparity d av In step S306, the road surface detection unit 26 uses the average road surface candidate parallax d av The road surface height Y in the mathematical formula (1) is changed. The road surface detection unit 26 calculates the geometrically estimated road surface parallax d using the mathematical formula (1) with the changed road surface height Y. s Furthermore, in order to extract the road surface candidate disparity d in the n+2th row, c The road surface detection unit 26 estimates the road surface parallax d in consideration of the geometry. s The parallax collection threshold is calculated based on the road height change ΔY in .

[0095] The road surface detection unit 26 calculates the road surface candidate disparity d c The extraction object is the candidate disparity d of the road surface closest to the stereo camera 10. c Shift upward (in the negative direction of the v coordinate) from the row corresponding to the extraction position of , and extract the candidate road disparity d corresponding to the v coordinate c The road surface detection unit 26 can extract the road surface candidate disparity d c The average road candidate disparity d corresponding to the corresponding u coordinate and v coordinate and the v coordinate av Stored together in the memory 23.

[0096] When the geometrically estimated road surface parallax d calculated by equation (1) in step S307 is s When the road surface candidate parallax d is equal to or less than the above-mentioned predetermined value, the road surface detection unit 26 ends the process of calculating the road surface candidate parallax d. c The extraction process and return to Figure 7 The predetermined value can be set to one pixel, for example.

[0097] As mentioned above, in Figure 8 In the flowchart, the road candidate disparity d is extracted c The initial value of the v coordinate is set to v0 corresponding to the position on the near side as viewed from the stereo camera 10, and the road surface candidate disparity d on the far side is sequentially extracted. c Generally speaking, the stereo camera 10 has a higher parallax detection accuracy at the short distance side than at the long distance side. Therefore, by sequentially extracting the road surface candidate parallax d from the short distance side to the long distance side, c , which can improve the detected road candidate disparity d c precision.

[0098] In the above extraction of the candidate road disparity d c of Figure 8 In the flowchart, the road candidate disparity d is calculated for each coordinate in the longitudinal direction. c In other words, when extracting the road candidate disparity d c In the flowchart, the road candidate disparity d is calculated for each row of pixels in the vertical direction. c The calculation unit of the road candidate disparity is not limited to this. The road candidate disparity d may also be calculated in units of multiple coordinates in the longitudinal direction. c .

[0099] The road surface candidate disparity d in steps S301 to S307 c After the extraction process, the road surface detection unit 26 enters Figure 7 The road surface detection unit 26 estimates the road surface parallax d in order from the short distance side to the long distance side. r When the road parallax d r The Kalman filter is applied sequentially. Therefore, the road surface detection unit 26 initializes the Kalman filter (step S202). As the initial value of the Kalman filter, the road surface parallax d calculated in step S305 can be used. r The average road surface candidate disparity d corresponding to the lowest row (row with v coordinate value v0) among the estimated rows av The value of .

[0100] The road surface detection unit 26 changes the target row from the short distance side to the long distance side of the road surface 41 and sequentially executes the following processes of steps S203 to S210 (step S203 ).

[0101] First, the road surface detection unit 26 detects the target line in the first parallax image based on the road surface candidate parallax d within a certain width range in the real space. c , generating the road parallax d r The range of a certain width in the actual space is a range that takes into account the width of the lane of the road. The certain width can be, for example, a value such as 2.5m or 3.5m. The range for obtaining the parallax is initially set to, for example, Figure 11 The range surrounded by the realization frame line 45 in the object detection device 20 is pre-stored in the memory 23 of the object detection device 20. By limiting the range of obtaining parallax to this range, the possibility of the road surface detection unit 26 mistakenly identifying objects other than the road surface 41 or structures such as sound insulation walls as the road surface 41 and extracting them is reduced. As a result, the accuracy of road surface detection can be improved. As will be described later, Figure 11 The parallax acquisition range indicated by the solid line in the figure can be changed sequentially from the initially set frame line 45 according to the situation on the road ahead.

[0102] The road surface detection unit 26 uses a Kalman filter to detect the road surface parallax d r The predicted value of the road surface parallax d is set for the target line r The acquisition range of road parallax d r The acquisition range is based on the Kalman filter to predict the road parallax d of the next line r The reliability is calculated based on the range of the reliability. The reliability is calculated based on the variance σ of the Gaussian distribution. 2 (σ is the road parallax d r The road surface detection unit 26 can calculate the road surface parallax d based on the predicted value ±2σ, etc. r The road surface detection unit 26 obtains the road surface candidate disparity d generated in step S204 from the road surface candidate disparity d c The histogram of the road disparity d is extracted in the Kalman filter-based setting r The road parallax d with the maximum frequency within the acquisition range r The road surface detection unit 26 extracts the road surface parallax d r Set the road parallax d of the target row r Observation value (step S205).

[0103] Next, the road surface detection unit 26 confirms the road surface parallax d determined in step S205. r is the correct road parallax d rThe road surface detection unit 26 generates a dv correlation map in which all road surface parallaxes d detected in each row up to the row currently being processed are converted into a dv correlation map. r Associated with the road parallax d r and v coordinates as the dv coordinate space of the coordinate axis. When the correct road surface 41 is detected, the dv correlation diagram is as follows Figure 12 As shown by the dotted line in the figure, as the value of v coordinate decreases, the road parallax d r It also decreases linearly.

[0104] On the other hand, when the parallax of the object is mistakenly recognized as the road surface 41, as shown in FIG. Figure 13 As shown in the dv correlation diagram, the parallax d at the portion representing the parallax of the object is almost constant regardless of the change in the longitudinal coordinate (v coordinate). Generally speaking, since the object includes a portion perpendicular to the road surface 41, it is displayed on the first parallax image as containing a large amount of parallax at equal distances. Figure 13 In the first portion R1, parallax d decreases as the v coordinate value changes. The first portion R1 represents the portion where the road surface 41 is correctly detected. In the second portion R2, parallax d remains constant even when the v coordinate changes. The second portion R2 is considered to be where an object is incorrectly detected. The road surface detection unit 26 can determine that an object has been incorrectly detected when a predetermined number of lines with approximately equal parallax d values persist.

[0105] If it is determined in step S206 that the parallax is not the correct road parallax d r (Step S206: No), the road surface detection unit 26 starts searching again for the road surface parallax d starting from the row where the object is determined to be erroneously recognized. r (Step S207). In step S207, the road surface detection unit 26 re-searches the road surface parallax histogram in the area of the row where the parallax d does not change even when the v coordinate value changes. If there is a high-frequency parallax in the area where the parallax is smaller than the parallax d determined in step S205, the road surface detection unit 26 can determine that the parallax is the correct road surface parallax d. r Observed values.

[0106] In step S206, the road parallax d r If it is judged to be correct (step S206: Yes) or in step S207, the road parallax d r When the re-search is completed, the road surface detection unit 26 proceeds to step S208. In step S208, the road surface detection unit 26 determines the lateral range of the road surface 41 on the first parallax image, which is the object of generating the next row of histograms that are offset by one pixel in the vertical direction. For example, Figure 11As shown, when another vehicle 42 exists on the road surface 41, the road surface detection unit 26 cannot obtain the correct road surface parallax d at the portion of the road surface 41 that overlaps with the other vehicle 42. r If the road parallax d can be obtained r The road surface 41 is narrowed, and it is difficult for the road surface detection unit 26 to obtain accurate road surface parallax d r Therefore, if Figure 11 As shown by the middle dotted line, the road surface detection unit 26 sequentially changes the number of pixels used to obtain the candidate road surface disparity d in the horizontal direction. c Specifically, when the road surface detection unit 26 determines that an object is included in step S206, it detects which side of the object in the lateral direction represents the correct road surface parallax d r The candidate disparity of the road surface d c In the next row, the range of obtaining parallax is gradually increased to have more parallax d indicating the correct road surface. r The candidate disparity of the road surface d c On one side (on Figure 11 Move to the middle right).

[0107] Then, the road surface detection unit 26 uses the road surface parallax d of the current line determined in step S205 or S207 to calculate the road surface parallax d of the current line. r To update the Kalman filter (step S209). That is, the Kalman filter is based on the road parallax d of the current row. r The observation value is used to calculate the road parallax d r When the estimated value of the current line is calculated, the road surface detection unit 26 calculates the road surface parallax d of the current line. r The estimated value of is added as part of the past data and used to calculate the road parallax d of the next line. r The height of the road surface 41 is assumed not to change suddenly relative to the horizontal distance Z from the stereo camera 10. Therefore, in the estimation using the Kalman filter in this embodiment, the road surface parallax d of the next row is estimated. r The road parallax d present in the current row r In this way, the road surface detection unit 26 limits the range of the histogram parallax used to generate the next row to the road surface parallax d of the current row. r The possibility of erroneously recognizing objects other than the road surface 41 is reduced. In addition, the amount of calculation performed by the road surface detection unit 26 can be reduced, thereby increasing the processing speed.

[0108] When the road surface parallax d estimated by the Kalman filter in step S209 r If the road parallax d estimated by the Kalman filter is greater than the predetermined value, the road surface detection unit 26 returns to step S203 and repeats the processing of steps S203 to S209. rIf the value is less than or equal to a predetermined value (step S210), the road surface detection unit 26 proceeds to the next process (step S211). The predetermined value may be, for example, one pixel.

[0109] In step S211, the road surface detection unit 26 maps the longitudinal image coordinate v and the estimated road surface parallax d on the dv correlation map. r The relationship between is approximated by two straight lines. r The value of the v coordinate is related to the distance z from the stereo camera 10 and the road surface height Y. Therefore, the v coordinate and the road surface parallax d r The relationship between the two straight lines can be considered to be the relationship between the distance to the stereo camera 10 and the height of the road surface 41 approximated by two straight lines. Figure 14 The flowchart of FIG. 1 illustrates the processing of step S211 in detail.

[0110] First, through the Figure 7 The process up to step S210 obtains the road parallax d r and the correlation between the v coordinate. For example, the v coordinate and the road parallax d r The correlation between Figure 15 The dashed line graph 51 is shown in the dv coordinate space. In real space, if the road surface 41 is flat and its inclination does not change, the graph 51 is a straight line. However, in the actual road surface 41, the inclination of the road surface 41 may change due to ups and downs. If the inclination of the road surface 41 changes, the graph 51 in the dv coordinate space cannot be represented by a straight line. If an attempt is made to approximate the change in the inclination of the road surface 41 using three or more straight lines or curves, the processing load of the object detection device 20 increases. Therefore, in this application, two straight lines are used to approximate the graph 51.

[0111] like Figure 15 As shown, the road surface detection unit 26 uses the first straight line 52 to calculate the estimated road surface parallax d on the lower side (short distance side) in the dv coordinate space by the least square method. r Approximation processing is performed (step S401). Approximation using the first straight line 52 enables the object detection device 20 to detect the object within the distance range up to the road surface parallax d corresponding to the predetermined distance. r The predetermined distance may be half the distance range within which object detection device 20 is to detect objects. For example, if object detection device 20 is designed to detect objects up to 100 meters ahead, first straight line 52 may be determined by the least squares method to be closest to figure 51 within a range from the shortest distance measurable by stereo camera 10 to 50 meters.

[0112] Next, the road surface detection unit 26 determines whether the inclination of the road surface 41 represented by the first straight line 52 approximated in step S401 is a possible inclination of the road surface 41 (step S402). The inclination angle of the first straight line 52 becomes a plane when converted to real space. The inclination of the first straight line 52 corresponds to the inclination angle of the road surface 41 in the YZ plane, determined based on conditions such as the road surface height Y0 at the installation location of the stereo camera 10 and the baseline length B. If the inclination of the road surface 41 in real space corresponding to the first straight line 52 is within a predetermined angle range relative to the horizontal plane in real space, the road surface detection unit 26 can determine that the inclination of the road surface 41 is a possible inclination. If the inclination of the road surface 41 in real space corresponding to the first straight line 52 is outside the predetermined angle range relative to the horizontal plane in real space, the road surface detection unit 26 can determine that the inclination of the road surface 41 is not a possible inclination. The predetermined angle can be appropriately set based on consideration of the driving environment of the mobile object 30.

[0113] If it is determined in step S402 that the inclination of the first straight line 52 is an inclination that cannot exist as the road surface 41 (step S402: No), the road surface detection unit 26 determines the first straight line 52 based on a theoretical road surface assuming that the road surface 41 is flat (step S403). The theoretical road surface can be calculated based on the installation conditions such as the road surface height Y0 at the installation position of the stereo camera 10, the installation angle, and the baseline length B. The road surface detection unit 26 can calculate the road surface parallax d calculated from the image. r For example, the road surface detection unit 26 may mistakenly use the parallax of an object or structure other than the road surface 41 as the road surface parallax d r In the case of extraction, it is determined that the road surface 41 has an inclination that cannot exist, thereby eliminating the error. In this way, it is possible to reduce the error of misjudging the parallax of objects or structures other than the road surface 41 as the road surface parallax d r possibility.

[0114] If, in step S402, the road surface detection unit 26 determines that the inclination of the first straight line 52 is a possible inclination of the road surface 41 (step S402: Yes), or after step S403, the road surface detection unit 26 proceeds to step S404. In step S404, the road surface detection unit 26 determines an approximation starting point 53 for approximating the second straight line 55. The road surface detection unit 26 calculates the approximation error with respect to the graph 51 from the smallest v-coordinate of the first straight line 52 (the far side) to the largest v-coordinate (the near side), and can select as the approximation starting point 53 the coordinate on the first straight line 52 where the approximation error continuously decreases to a predetermined value. Alternatively, the road surface detection unit 26 can calculate the approximation error with respect to the graph 51 from the largest v-coordinate (the near side) to the smallest v-coordinate (the far side), and select as the approximation starting point 53 the coordinate on the first straight line 52 where the approximation error exceeds a predetermined value. The v-coordinate of the approximation starting point 53 is not fixed to a specific value. Approximation starting point 53 can be set on first straight line 52 at a position corresponding to a v-coordinate closer to the stereo camera 10 than half the distance range within which object detection device 20 performs object detection. For example, when first straight line 52 approximates road surface 41 within a range from the closest measurable distance to 50 meters ahead, approximation starting point 53 can be set at a position corresponding to a v-coordinate 40 meters ahead of 50 meters.

[0115] After step S404, the road surface detection unit 26 repeatedly executes steps S405 to S407. Figure 16 As shown, the road surface detection unit 26 uses the angle difference from the first straight line 52 as an angle selected from a predetermined angle range to sequentially select candidates for the second straight line 55 starting from the approximation starting point 53 (step S405). The predetermined angle range is set to the angle within which the road slope can change within the distance range of the measurement target. The predetermined angle range can be, for example, ±3 degrees. For example, the road surface detection unit 26 may increment the angle of the candidate straight line 54 by 0.001 degrees, starting from -3 degrees from the angle of the first straight line 52 to +3 degrees from the angle of the first straight line 52.

[0116] For each selected candidate straight line 54, the road surface detection unit 26 calculates the error between the selected candidate straight line 54 and the upper (far-distance) portion of the approximate starting point 53 of the graph 51 in the dv coordinate space (step S406). The error can be calculated as the mean square error of the parallax d relative to the v coordinate. The road surface detection unit 26 may store the error calculated for each candidate straight line 54 in the memory 23.

[0117] After the road surface detection unit 26 completes the error calculation for all candidate straight lines 54 within the angle range (step S407), it searches for the minimum error among the errors stored in the memory 23. Figure 17 As shown, the road surface detection unit 26 selects the candidate straight line 54 having the smallest error as the second straight line 55 (step S408).

[0118] When the second straight line 55 is determined in step S408, the road surface detection unit 26 determines whether the error between the second straight line 55 and the graph 51 is within a predetermined value (step S409). The predetermined value may be appropriately set to obtain desired road surface estimation accuracy.

[0119] In step S409, if the error is within the predetermined value (step S409: Yes), the road parallax d r Approximation processing is performed using the first straight line 52 and the second straight line 55 .

[0120] In step S409, if the error exceeds the predetermined value (step S409: No), the road surface detection unit 26 extends the first straight line 52 upward (far distance side) and rewrites the approximation result (step S410). r Use two straight lines for approximation.

[0121] The road parallax d relative to the v coordinate is approximated by using two straight lines r , the road surface shape is approximated by two straight lines. This reduces the subsequent computational load and speeds up object detection compared to approximating the road surface shape using a curve or three or more straight lines. Furthermore, the error with the actual road surface is smaller than when the road surface is approximated using a single straight line. Furthermore, by not fixing the v coordinate of the approximation starting point 53 of the second straight line 55 at a predetermined coordinate, the accuracy of the approximation with the actual road surface can be improved compared to when the coordinates of the approximation starting point 53 are fixed in advance.

[0122] In step S409, if the error is within the predetermined value (step S409: Yes) or after step S410, the road parallax d r The linear approximation process is completed and returns to Figure 7 Step S212.

[0123] In step S212, the road parallax d removed from the first parallax image is determined. r The road parallax d removed from the first parallax image is r The threshold value corresponds to the first height described later. The first height can be calculated so that the road parallax d is removed in the subsequent processing of step S103. r .

[0124] Then, the processing of the image processing unit 22 returns to Figure 5The unnecessary parallax removal unit 27 of the image processing unit 22 obtains from the road surface detection unit 26 the image of the v coordinate in the dv coordinate space and the road surface parallax d approximated by two straight lines. r The approximate formula of the relationship between v coordinate in dv coordinate space and road parallax d r The relationship between the front distance Z of the stereo camera 10 and the road surface height Y in the real space can be obtained by using an approximate expression for the relationship. The unnecessary parallax removal unit 27 performs the second process (step S103) based on the approximate expression. The second process is to remove the parallax pixels corresponding to the subject whose height from the road surface 41 is less than the first height and the parallax pixels corresponding to the subject whose height from the road surface 41 is more than the second height in the real space from the first parallax image. In this way, the unnecessary parallax removal unit 27 removes the parallax pixels corresponding to the subject whose height from the road surface 41 is less than the second height in the real space. Figure 6 The first parallax image shown is generated Figure 18 The second parallax image is shown. Figure 18 The actual second parallax image based on the image acquired from the stereo camera 10 is as shown in FIG. Figure 19 As shown. Figure 19 In the image, the size of the parallax is expressed by the shades of black and white.

[0125] The first height can be set to, for example, a value greater than 15 cm and less than 50 cm. If the first height is set to less than 15 cm, the parallax removal unit 27 may easily detect subjects other than objects on the road due to factors such as unevenness of the road surface 41 and / or errors in the approximation formula. This may result in detection errors or a decrease in detection speed. Furthermore, if the first height is set to greater than 50 cm, the parallax removal unit 27 may fail to detect children and / or large obstacles on the road surface 41.

[0126] The second height is set based on the upper limit of vehicle heights permitted on the road. This permitted vehicle height is regulated by traffic laws. For example, under Japan's Road Traffic Law, truck heights are generally limited to 3.8 meters or less. For example, the second height could be 4 meters. If the second height is higher than 4 meters, unnecessary objects, such as overhead structures like traffic lights and information display boards, may be detected.

[0127] By removing unnecessary parallax, the unnecessary parallax removal unit 27 can pre-remove parallax for subjects other than road surface 41 or objects on the road before the subsequent object detection processes (third and fourth processes) for detecting objects. This improves the accuracy of object detection. Furthermore, by removing unnecessary parallax, the amount of computation associated with parallax unrelated to objects on the road can be reduced, thereby increasing processing speed. Therefore, the object detection device 20 of the present disclosure can improve the performance of the object detection process.

[0128] The unnecessary parallax removal unit 27 transmits the second parallax image to the clustering unit 28. Based on the second parallax image and the shape approximation of the road surface 41 calculated by the road surface detection unit 26, the clustering unit 28 performs a third process (step S104) to determine the object parallax associated with the object for each horizontal u-coordinate or range of u-coordinates. Specifically, the clustering unit 28 generates a histogram representing the number of pixels for each parallax for each horizontal u-coordinate range in the second parallax image. A u-coordinate range is a range that includes one or more horizontal pixels.

[0129] like Figure 20 As shown, the clustering unit 28 extracts a longitudinal region having one or more horizontal pixel widths Δu from the two-parallax image. The longitudinal region is used to determine whether there is an object parallax associated with the object in the region and to obtain distance information corresponding to the object parallax. Therefore, if the horizontal width Δu of the longitudinal region is made thinner, the resolution of detecting horizontal objects becomes higher. Figure 20 For ease of explanation, the width of Δu is shown as relatively wide, but Δu can be anywhere from one pixel to several pixels. Vertical regions are sequentially acquired from one horizontal end of the second parallax image to the other, and the following processing is performed to detect object parallax across the entire horizontal area of the second parallax image.

[0130] Figure 21 It is a diagram showing an example of a histogram. Figure 21 The horizontal axis is the disparity d expressed in pixels. The disparity d is larger on the left side of the horizontal axis and gradually decreases towards the right. The minimum value of the disparity d can be, for example, one pixel or less. The larger the disparity, the finer the resolution of the distance, and the smaller the disparity, the coarser the resolution of the distance. Therefore, Figure 21 The horizontal axis of the histogram gathers more disparity d on the side with larger disparity. Figure 21 The vertical axis of the histogram represents the number of occurrences of disparity pixels with the horizontal axis representing disparity d.

[0131] exist Figure 21 In FIG, a threshold curve is also shown, which represents a threshold value for determining whether each disparity d is an object disparity. Disparity d can be a representative value of an interval of disparity having a width. When the number of occurrences of pixels of each disparity d exceeds the threshold curve, it means that a predetermined number or more of disparity pixels representing the same distance specified by the threshold curve are included in a longitudinal area of width Δu. Figure 21In the figure, the area with slashes (columnar portion) exceeds the threshold curve. The threshold curve can be set to the number of occurrences (number of pixels) corresponding to a predetermined height in the y direction in the actual space for each parallax d. For example, the predetermined height can be 50 cm. When the parallax is small at a long distance, the object displayed on the image is smaller than the object at a close distance. Therefore, as the parallax becomes smaller, the value of the number of occurrences on the vertical axis of the threshold curve becomes smaller. When the number of occurrences exceeds the predetermined threshold corresponding to the parallax d, the clustering unit 28 determines the parallax d corresponding to the pixel as the object parallax d. e .

[0132] Next, the clustering unit 28 classifies the object based on the parallax d e The parallax d with the object is calculated based on the distribution of the road surface 41 and the shape of the road surface 41 estimated in step S102. e Associated height information (step S105). The process of step S105 may be included in the third process.

[0133] Specifically, it is assumed that the image acquired by the first camera 11 or the second camera 12 contains the following Figure 22 The portion of the second parallax image corresponding to the portion surrounded by the frame line 61 of the other vehicle 42 is shown in FIG. Figure 23 Shown enlarged.

[0134] The clustering unit 28 is for objects with parallax d e The u coordinate is based on the object parallax d e The distance information and the road shape estimated in step S102 are used to calculate the parallax d with the object. e When there is an object on the road surface 41, there is an object parallax d e The parallax pixels of d are arranged above the estimated road surface position on the second parallax image. The clustering unit 28 scans the parallax pixels of the second parallax image upward from the road surface position of the u coordinate to detect the object with parallax d. e The clustering unit 28 is based on the distribution of the disparity pixels of the same object in the vertical direction (v coordinate direction). e The height information on the second disparity image is determined by the number or distribution of the disparity pixels arranged. e Even when the parallax pixels are partially interrupted in the vertical direction, the clustering unit 28 can determine the height information based on a predetermined determination criterion.

[0135] The clustering unit 28 may associate the object disparity d with respect to each range of the horizontal coordinate (u coordinate) including one or more coordinates. e and height information, and stores it in the memory 23. Figure 24As shown in one example, the clustering unit 28 can store multiple object disparities d e It is expressed as a point cloud distribution in a two-dimensional space (ud coordinate space) with the u coordinate and the parallax d as the horizontal and vertical axes.

[0136] The clustering unit 28 classifies the object disparity d at each u coordinate into e The information of is sent to the grouping unit 29. The grouping unit 29 is divided into two groups: e The information is converted into a coordinate system in the real space composed of xz coordinates, and the object parallax d is extracted e Set (group), thereby performing the process of detecting the object, that is, the fourth process (step S106). Figure 25 and Figure 26 An example of processing performed by the grouping unit 29 will be described.

[0137] exist Figure 25 In FIG, a moving body 30 equipped with the object detection system 1 and other vehicles 42 are included, which are traveling on a road surface 41. Figure 25 In FIG, the moving object 30 is a vehicle.

[0138] The grouping unit 29 of the object detection device 20 mounted on the moving body 30 Figure 24 The disparity d of multiple objects in the ud coordinate space shown e Convert to Figure 26 The point cloud in the actual space (xz coordinate space) is shown. Figure 26 In the ud coordinate space, the object disparity d is represented e Each point is represented as a point in the xz coordinate space. The grouping unit 29 extracts a set of point clouds based on the distribution of the point clouds. The grouping unit 29 collects a plurality of similar points according to a predetermined condition and extracts them as a set of point clouds. The set of point clouds represents the object parallax d e A collection (group) of .

[0139] When an object has a surface parallel to the baseline length direction of the stereo camera 10, the point cloud is arranged along the x direction in the xz coordinate space. When a set 71 of point clouds arranged along the x direction exists in the xz coordinate space, the grouping unit 29 can recognize it as an object. Figure 26In the example, point cloud set 71 corresponds to the back of another vehicle 42. If point cloud set 72 is present in the xz coordinate space and arranged along the z direction, grouping unit 29 can identify it as a roadside structure such as a guardrail or the side of another vehicle 42. Point cloud set 72 arranged along the z direction in the xz coordinate space corresponds to an object aligned parallel to the direction of travel of mobile object 30 or a plane of an object parallel to the direction of travel of mobile object 30. Grouping unit 29 can exclude point clouds 72 aligned along the z direction from the object detection process.

[0140] The grouping unit 29 can detect the width of the object based on the width of the set of point clouds 71 identified as the object arranged in the x direction. The grouping unit 29 can detect the width of the object based on the object disparity d obtained by the clustering unit 28 in step S105. e The height of the object is determined by the associated height information. Therefore, the grouping unit 29 can identify the position, horizontal width and height of the identified object in the xz coordinate space.

[0141] The image processing unit 22 can output the position, lateral width, and height information of the object identified by the grouping unit 29 to other devices in the mobile body 30 through the output unit 24 (step S107). For example, the image processing unit 22 can output this information to a display device in the mobile body 30. Figure 27 As shown, the display device in the mobile body 30 can display a frame line surrounding the image of the other vehicle 42 on the image of the first camera 11 or the second camera 12 based on the information obtained from the object detection device 20. Figure 27 In the figure, the frame lines indicate the position of the detected objects and the range they occupy in the image.

[0142] As described above, the object detection device 20 of the present disclosure can achieve fast processing speed and high-precision object detection. In other words, the object detection device 20 of the present disclosure can improve object detection performance. Furthermore, the object detection device 20 does not limit the objects to be detected to specific types. The object detection device 20 can detect any object on the road surface. The image processing unit 22 of the object detection device 20 can perform the first, second, third, and fourth processes without using information from the images captured by the stereo camera 10 other than the first parallax image. Therefore, the object detection device 20 does not need to perform processing to separately identify objects from the captured images, in addition to processing the first and second parallax images. Consequently, the object detection device 20 of the present disclosure can reduce the processing load on the image processing unit 22 involved in object recognition. This does not preclude the object detection device 20 of the present disclosure from being combined with image processing performed directly on images obtained from the first camera 11 or the second camera 12. The object detection device 20 can also be combined with image processing techniques such as template matching.

[0143] In the above description of the processing performed by the image processing unit 22, to facilitate understanding of the present disclosure, the processing includes determinations and operations using various images. This processing using images does not necessarily include actual image rendering. Processing having substantially the same content as that using these images is performed through information processing within the image processing unit 22.

[0144] Although the embodiments according to the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various deformations or modifications based on the present disclosure. Therefore, it should be noted that these deformations or modifications are included in the scope of the present disclosure. For example, the functions included in each component or each step, etc. can be reconfigured in a logically non-contradictory manner, and multiple components or steps, etc. can be combined into one or divided. Although the embodiments according to the present disclosure are described with the device as the center, the embodiments according to the present disclosure can also be implemented as a method including steps performed by each component of the device. The embodiments according to the present disclosure can also be implemented as a method, a program, or a storage medium with a program recorded thereon, which is executed by a processor included in the device. It should be understood that the scope of the present disclosure also includes these.

[0145] In the present disclosure, the descriptions such as "first" and "second" are identifiers for distinguishing the configurations. The configurations distinguished by the descriptions such as "first" and "second" in the present disclosure can have their numbers swapped. For example, "first," which is an identifier for the first lens, can be replaced with "second," which is an identifier for the second lens. The exchange of identifiers is performed simultaneously. The configurations can be distinguished even after the identifiers are swapped. Identifiers can be deleted. Configurations from which identifiers have been deleted are distinguished by reference numerals. The descriptions based solely on identifiers such as "first" and "second" in the present disclosure cannot be used to explain the order of the configurations or to justify the existence of identifiers with smaller numbers.

[0146] In the disclosure, the x-direction, y-direction, and z-direction are provided for ease of description and are interchangeable. The components of the present disclosure are described using an orthogonal coordinate system with the x-direction, y-direction, and z-direction as the directions of the respective axes. The positional relationship of the components of the present disclosure is not limited to being in an orthogonal relationship. The u-coordinate and v-coordinate representing the coordinates of the image are provided for ease of description and are interchangeable. The origin and direction of the u-coordinate and the v-coordinate are not limited to those described in the disclosure.

[0147] In the above embodiment, the first camera 11 and the second camera 12 of the stereo camera 10 are arranged side by side in the x-direction. The arrangement of the first camera 11 and the second camera 12 is not limited to this. The first camera 11 and the second camera 12 may be arranged side by side in a direction perpendicular to the road surface (y-direction) or in a direction inclined relative to the road surface 41. The number of cameras constituting the stereo camera 10 is not limited to two. The stereo camera 10 may include three or more cameras. For example, by using two cameras arranged horizontally on the road surface and two cameras arranged vertically, i.e., a total of four cameras, more accurate distance information can be obtained.

[0148] In the above embodiment, the stereo camera 10 and the object detection device 20 are mounted on the mobile object 30. However, the stereo camera 10 and the object detection device 20 are not limited to being mounted on the mobile object 30. For example, the stereo camera 10 and the object detection device 20 may be mounted on roadside equipment installed at an intersection, etc., and configured to capture images including the road surface. For example, the roadside equipment may detect a first vehicle approaching from one of the intersecting roads at the intersection and provide a notification of the first vehicle's approach to a second vehicle approaching on the other road.

[0149] Captions

[0150] 1: Object Detection System

[0151] 10: Stereo Camera

[0152] 11: First Camera

[0153] 12: Second Camera

[0154] 20: Object detection device

[0155] 21: Get Department

[0156] 22: Image processing unit (processor)

[0157] 23: Memory

[0158] 24: Output

[0159] 25: Parallax image generation unit

[0160] 26: Road surface detection unit (road surface detection device)

[0161] 27: Unnecessary parallax removal unit

[0162] 28: Clustering Department

[0163] 29: Grouping Department

[0164] 30: Mobile

[0165] 41: Road surface

[0166] 42: Other vehicles (objects)

[0167] 45: Frame line

[0168] 51: Graphics

[0169] 52: First straight line

[0170] 53: Approximate starting point

[0171] 54: Candidate line

[0172] 55: Second straight line

[0173] 61: Frame line

[0174] 71, 72: Collection of point clouds

[0175] R1: Part 1

[0176] R2: Part 2

Claims

1. A road surface detection device, wherein: A processor is provided, the processor being configured to execute the following processing: acquiring or generating a first disparity map, the first disparity map being generated based on output from a stereo camera that captures an image including a road surface, and wherein the first disparity map is a map associating disparities obtained from the output of the stereo camera with two-dimensional coordinates consisting of a first direction and a second direction intersecting the first direction, wherein the first direction corresponds to a horizontal direction of the image captured by the stereo camera, The relationship between the coordinates of the road surface in the second direction included in the first disparity map and the disparity representing the road surface is approximated using two straight lines. The two straight lines include a first straight line and a second straight line, The processor approximates a near side in real space from a side where the stereo camera is located using the first straight line, and approximates a far side compared to the first straight line using the second straight line, The processor is configured to determine, based on the first disparity map, coordinates of an approximate starting point for starting approximation of the second straight line on the far side of the first straight line, The processor determines the coordinates of the approximate starting point based on an error between the first straight line and a parallax representing the road surface.

2. The road surface detection device according to claim 1, wherein: The processor determines the first straight line by approximating a portion from a side where the stereo camera is located to a coordinate in the second direction corresponding to a predetermined distance with a straight line.

3. The road surface detection device according to claim 2, wherein: The processor sets the predetermined distance to half of a distance range for performing object detection.

4. The road surface detection device according to claim 2 or 3, wherein: The processor determines the coordinates of the approximate starting point in the second direction as coordinates corresponding to a position closer to one side of the stereo camera in the real space than the predetermined distance.

5. The road surface detection device according to claim 2 or 3, wherein: The processor selects a straight line having the smallest error with the parallax representing the road surface as the second straight line from a plurality of second straight line candidates, wherein the plurality of second straight line candidates are straight lines starting from the approximate starting point and having an angle difference with the first straight line of an angle selected from a predetermined angle range.

6. An object detection device, wherein: A road surface detection device according to claim 1, The processor is configured to detect an object using a parallax representing the road surface approximated by the straight line.

7. The object detection device according to claim 6, wherein: The processor detects an object based on a second disparity map obtained by removing the disparity representing the road surface from the first disparity map.

8. An object detection system, wherein: have: A stereo camera captures multiple images with parallax differences between them. An object detection device comprising at least one processor, The processor is configured to acquire or generate a first disparity map, the first disparity map being generated based on an output of a stereo camera that captures an image including a road surface, and the first disparity map being a map associating disparities obtained from the output of the stereo camera with two-dimensional coordinates consisting of a first direction and a second direction intersecting the first direction, wherein the first direction corresponds to a horizontal direction of the image captured by the stereo camera, approximating a relationship between the coordinates of the road surface in the second direction included in the first disparity map and the disparity representing the road surface using two straight lines, and detecting an object using the disparity representing the road surface approximated by the straight lines. The two straight lines include a first straight line and a second straight line, The processor approximates a near side in real space from a side where the stereo camera is located using the first straight line, and approximates a far side compared to the first straight line using the second straight line, The processor is configured to determine, based on the first disparity map, coordinates of an approximate starting point for starting approximation of the second straight line on the far side of the first straight line, The processor determines the coordinates of the approximate starting point based on an error between the first straight line and a parallax representing the road surface.

9. A mobile object, wherein: A device comprising the object detection system according to claim 8.

10. A method for object detection, wherein: acquiring or generating a first disparity map, the first disparity map being generated based on output from a stereo camera that captures an image including a road surface, and wherein the first disparity map is a map associating disparities obtained from the output of the stereo camera with two-dimensional coordinates consisting of a first direction and a second direction intersecting the first direction, wherein the first direction corresponds to a horizontal direction of the image captured by the stereo camera, The relationship between the coordinates of the second direction of the road surface included in the first disparity map and the disparity representing the road surface is approximated using two straight lines, the two straight lines including a first straight line and a second straight line. The near side in the real space from the side where the stereo camera is located is approximated by the first straight line, and the far side compared to the first straight line is approximated by the second straight line, determining the coordinates of an approximate starting point for approximating the second straight line on the far side of the first straight line based on the first disparity map, determining the coordinates of the approximate starting point based on an error between the first straight line and a parallax representing the road surface, An object is detected using a parallax representing the road surface approximated by the first straight line and the second straight line.

Images