Environment recognition system

Abstract

To provide an environment recognition system capable of detecting road surface irregularities in front of a vehicle with high accuracy.SOLUTION: An environment recognition system according to the present invention is a system for recognizing a three-dimensional shape of a road surface on which an own vehicle travels. The own vehicle includes: an image processing unit that performs image processing to detect structures on the road surface from images captured by a vehicle-mounted camera; a vehicle communication unit that receives location information of the structures on the road surface from an external server; and an image processing method determination unit that changes an image processing method of the image processing unit based on the location information of the structures on the road surface received by the vehicle communication unit.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to an external recognition system for recognizing the unevenness of the road surface on which a vehicle travels.

Background Art

[0002] Conventionally, there has been an external recognition technology that images the front of a vehicle using an in-vehicle camera to recognize structures on the road surface such as speed bumps. And a system has been proposed that uses the external recognition information acquired by other vehicles for the driving control of the host vehicle (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When detection is performed by an external recognition sensor such as an in-vehicle camera, the unevenness of the road surface such as structures on the road surface may not be accurately recognized depending on environmental conditions such as at night, backlight, or rainy weather. Further, when using the external recognition information acquired by other vehicles for the driving control of the host vehicle, information on the accurate position of the structures on the road surface is required. However, position detection means such as GPS is easily affected by the surrounding environment, the detection accuracy is unstable, and there are also limitations in the detection accuracy.

[0005] The present invention has been made in view of the above points, and an object thereof is to provide an external recognition system capable of detecting the unevenness of the road surface in front of the vehicle with high accuracy.

Means for Solving the Problems

[0006] The external recognition system of the present invention for solving the above problems is an external recognition system that recognizes the three-dimensional shape of the road surface on which the host vehicle travels, The host vehicle includes an image processing unit that performs image processing to detect structures on the road surface from captured images captured by an in-vehicle camera, a vehicle communication unit that receives position information of the structures on the road surface from an external server, and an image processing method determination unit that changes the image processing method of the image processing unit based on the position information of the structures on the road surface received by the vehicle communication unit. It is characterized by comprising the above.

Advantages of the Invention

[0007] According to the present invention, road surface unevenness in front of the vehicle can be detected with high accuracy. Further features related to the present invention will become apparent from the description in this specification and the attached drawings. Also, problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Mode for Carrying Out the Invention

[0009] Next, embodiments of the present invention will be described with reference to the drawings.

[0010] First, the outline of the external environment recognition system of this embodiment will be described. FIG. 1 is a diagram showing the overall outline of the external environment recognition system. The external environment recognition system in this embodiment aggregates the information on the road surface conditions detected by at least one or more information-providing vehicles C1 in the server NS which is an external server, distributes the information on the road surface conditions around the information-using vehicle C2 from the server NS to the information-using vehicle C2, the information-using vehicle C2 further detects the road surface conditions in more detail using the information on the road surface conditions distributed from the server NS, and performs vehicle control based on the detection result.

[0011] The external recognition system utilizes a cloud-based map platform 101 that accumulates, manages, and analyzes the experience information necessary for the automatic driving of a vehicle on a map. The map platform 101 is what is referred to as traffic experience information on the map, and manages in a two-layer state the map information including roads, lanes, driving routes, facilities and utensils such as signs and traffic lights installed on the road, etc., which is called a road-related map, and the driving experience-related information in the past such as a drive history including driving experience information, with the two being associated with each other. The road-related map contains high-precision map information used for the automatic driving of a vehicle. The driving experience information includes information based on actual driving experience such as the frequency and time zone of traffic jams occurring on a predetermined road, and the average driving speed.

[0012] Vehicle information such as lane selection and appropriate speed is input into the map platform 101 from a plurality of information-providing vehicles C1 (C1a - C1n), and traffic information such as signals and traffic jams is input from the infrastructure 111. The map platform 101 uses a server NS connected to an information communication network and a server database DBs that stores various information in a readable manner to analyze big data including vehicle information and traffic information, generate driving experience information, and perform mapping on the road-related map.

[0013] A plurality of information-providing vehicles C1 and information-using vehicles C2 each use a self-position measurement device such as GNSS (Global Navigation Satellite System) to acquire information on their self-position in the world coordinate system (latitude and longitude coordinate system). The information-providing vehicle C1 performs external recognition using a three-dimensional measurement device, which is an external recognition device mounted on the vehicle, and transmits the information on the road surface conditions obtained through the external recognition to the server NS.

[0014] The information-using vehicle C2 performs more accurate automatic driving by receiving the distribution of traffic experience information on the map from the server NS of the map platform 101 based on its own position information. The information-using vehicle C2 includes a three-dimensional measurement device 124, an AD_ECU 122 that performs vehicle control such as automatic driving control using the three-dimensional external information acquired by the three-dimensional measurement device 124 and the information acquired from the server NS of the map platform 101, an MPU 123 that performs various arithmetic processes, and a steering device 125, a brake device 126, and a drive device 127 that are controlled by control signals from the AD_ECU 122.

[0015] FIG. 2 is a diagram showing an example of a scene in which a road surface structure is detected using the external recognition system in the present embodiment.

[0016] In the external recognition system, information aggregation processing and information distribution processing are performed. In the information aggregation processing, when a road surface structure is detected during the travel of the information-providing vehicle C1, the position information of the road surface structure ST is transmitted to the server NS, and the information is aggregated by the server NS. FIG. 2(a) is a diagram showing an example of information aggregation, and shows a scene in which a road surface structure ST is detected during daytime travel. When the three-dimensional measurement device of the information-providing vehicle C1 is, for example, a stereo camera, the road surface structure ST can be easily and accurately detected during travel in a bright time zone during the day.

[0017] In the information-providing vehicle C1, the road surface unevenness, which is the three-dimensional shape of the road surface, is calculated by the stereo camera, and the detection of the road surface structure ST and the estimation of the relative position and height (shape) of the road surface structure ST with respect to the information-providing vehicle C1 are performed. Then, the own position information of the information-providing vehicle C1 measured by the own position measurement device 315 and the position information of the road surface structure ST are transmitted to the server NS by the communication device and stored in the server database DBs as driving experience information. The position information of the road surface structure ST is stored separately for each road surface structure ST.

[0018] Position information of the road surface structures ST is transmitted to the server NS from a plurality of information providing vehicles C1. The server NS aggregates the position information of the road surface structures ST received from the plurality of information providing vehicles C1, and recalculates the position information for each road surface structure ST. In this recalculation, processing is performed to reduce the error of the position information by statistical processing and improve the accuracy of the position information.

[0019] In the information distribution process, by transmitting its own position information from the information using vehicle C2 to the server NS, information about the road surface conditions around the information using vehicle C2 is distributed from the server NS to the information using vehicle C2. FIG. 2(b) is a diagram showing an example of information distribution, and shows a scene of detecting a road surface structure during night driving. The stereo camera of the information using vehicle C2 may have a low success rate and detection accuracy in detecting the road surface structure ST, which is the object to be detected, during driving in the dark hours of the night.

[0020] On the other hand, in the present embodiment, when the information using vehicle C2 receives the distribution of the position information of the road surface structures existing around its own position from the server NS, it determines the image processing method of the image processing unit for detecting the road surface structures based on the position information by the stereo camera, and detects the road surface structure ST by the determined image processing method. For example, when more accurate position information of the road surface structure ST is obtained from the server NS, changes in the detection algorithm such as narrowing the area for image processing by the stereo camera to perform high-precision image recognition, or changes in the detection parameters such as narrowing the range for detecting the peak height position of the road surface unevenness and increasing the threshold value are performed. Further, when the information using vehicle C2 cannot detect the road surface structure ST with the stereo camera or the detection accuracy is low due to weather, external light conditions, etc., the information using vehicle C2 does not use the recognition result by image recognition, and performs vehicle control such as warning to the occupant, suspension pressure, deceleration, and steering using the position information of the road surface structure ST distributed from the server NS.

[0021] FIG. 3 is a functional block diagram of the information providing vehicle C1, the information using vehicle C2, and the server NS that constitute the external environment recognition system of the present embodiment. The information-providing vehicle C1 includes a stereo camera 312 which is an example of a three-dimensional measurement device that measures the three-dimensional shape of the driving road surface and detects structures on the road surface, a G-sensor 313 that detects vehicle vibrations applied to the information-providing vehicle C1 by passing over the structure ST on the road surface, a self-position measurement device (position information acquisition means) 315 that measures its own coordinate position in the world coordinate system such as a GPS navigation device, and a vehicle-road communication device 317 that transmits and receives information to and from the server NS.

[0022] The information-utilizing vehicle C2 includes a stereo camera 322, a self-position measurement device (self-position measurement unit) 325, a vehicle-road communication device 324, and a G-sensor 323. Further, it has a vehicle control device 326 that performs automatic braking control and inter-vehicle distance control ACC based on the detection results of the stereo camera 322, a human-machine interface 327 that alerts the vehicle occupants, a vehicle route generation device 328 that generates a route to the destination, and a vehicle database DBc that stores information such as the coordinate position of the information-utilizing vehicle C2, the position and size of the structure ST on the road surface. The stereo camera 322 includes a pair of in-vehicle cameras that image the front of the vehicle and a camera control unit that controls the in-vehicle cameras and image processing of the captured images. The camera control unit has hardware such as a CPU and a memory, and software programs executed by the hardware, and embodies the control functions of the following parts through the cooperation of these hardware and software. The camera control unit of the stereo camera 322 includes an image processing unit that performs image processing to detect structures on the road surface from the captured images captured by the in-vehicle cameras, and an image processing method determination unit that changes the image processing method of the image processing unit based on the position information of the structure ST on the road surface received from the server NS.

[0023] The server NS has a communication device 302 that transmits and receives information between the vehicle-to-vehicle communication device 317 of the information-providing vehicle C1 and the vehicle-to-vehicle communication device 324 of the information-using vehicle C2, a server database DBs that aggregates and stores in a readable manner the position information of the road surface structures ST provided from a plurality of information-providing vehicles C1, and a data analysis device 303 that performs statistical analysis of big data including vehicle information and traffic information stored in the server database DBs, generates driving experience information, and maps it on a road-related map.

[0024] Figure 4 is a flowchart for explaining the content of information processing in the information-providing vehicle and the server, and shows the content of the process of providing information from the information-providing vehicle C1 to the server NS.

[0025] In the information-providing vehicle C1, the imaging unit of the stereo camera 312 captures the front of the vehicle to obtain a captured image (S401), the image processing unit of the stereo camera 312 performs image processing on the captured image to generate a disparity image, measures the road surface unevenness which is the three-dimensional shape in front of the vehicle, and detects the road surface structure ST based on the measurement result (S402). The stereo camera 312 obtains information on the three-dimensional shape such as speed bumps on the road surface, the shape of the road edge, or road surface unevenness data, and the position of the road surface structure ST through the image processing of the image processing unit.

[0026] The image processing unit of the stereo camera 312 converts the coordinates of the position of the road surface structure ST obtained by image processing from the relative position to the vehicle to the coordinate position in the world coordinate system. Specifically, it obtains the position information in the world coordinate system corresponding to the imaging position of the vehicle measured by the self-position measurement device 315 (S403), and generates the position information of the road surface structure ST in the world coordinate system (S404). Then, the information-providing vehicle C1 transmits the position information of the road surface structure ST after coordinate conversion to the server NS through the vehicle-to-vehicle communication device 317.

[0027] Server NS stores the position information of a plurality of road surface structures ST transmitted from a plurality of information - providing vehicles C1 or from the same information - providing vehicle C1 in the server database DBs (S405). When Server NS stores the position information of the road surface structure ST in the server database DBs, it recalculates the traffic experience information on the map at a predetermined timing and updates it to the latest information. When Server NS receives the position information of a plurality of road surface structures ST from the information - providing vehicle C1, it performs sorting such as addition and deletion of information, aggregates the information at the same position as the same information, and stores it in the server database DBs. Server NS statistically analyzes the position information of the road surface structures ST transmitted from a plurality of information - providing vehicles C1 and stored in the server database DBs as big data by the data analysis device 303, creates and updates driving experience information, maps it onto the road - related map, and stores it in the server database DBs as traffic experience information on the map (S406).

[0028] FIG. 5 is a flowchart for explaining a normal image - processing method in a stereo camera, and shows an example of the processing performed by the stereo camera of the information - providing vehicle C1.

[0029] First, a pair of left - and - right captured images are captured in the imaging unit of the stereo camera 312 (S501), and a disparity image is generated from this pair of captured images (S502). Then, an image - processing area for performing image processing within the disparity image is set (S503). In this embodiment, in order to acquire road surface unevenness data, the area on the tire traveling path through which the left and right tires of the information - providing vehicle C1 pass is set as the image - processing area.

[0030] FIG. 8 is a diagram showing an example of setting an image processing area in a disparity image and is a diagram for explaining a method of setting the image processing area. In the disparity image 801 generated by imaging the front of the vehicle with a stereo camera, a driving surface 811 of a road R0 and white lines 812 are shown on both the left and right sides thereof. Then, processing areas 821 and 822 are set along the vehicle traveling path. The processing areas 821 and 822 are set so as to extend in the depth direction (traveling direction) with a predetermined width on the tire traveling path through which the left and right tires pass as the vehicle traveling path.

[0031] Next, a disparity detailed analysis for each horizontal search is performed (S504). Here, within the processing areas 821 and 822 of the disparity image 801, as shown by arrows 831 and 832 in FIG. 8, a search is made in the horizontal direction of the image. Then, as shown in FIG. 10, a disparity value at which the number of votes becomes the most frequent value is adopted. FIG. 10 is a diagram for explaining a method of determining the disparity of road surface unevenness by searching in the horizontal direction. In the example shown in FIG. 8, the horizontal search is performed over the entire depth direction of the processing areas 821 and 822.

[0032] Next, the disparity values for each horizontal search calculated in S504 are arranged in the depth direction of the processing area, and a process of calculating the peak position of the unevenness height of the road surface is performed (S505). For example, a peak position having a value larger than a preset determination threshold value for the height of the peak position from the road surface reference plane can be determined as the position of the road surface structure ST.

[0033] Then, a process of analyzing the stability of the peak position determination is performed (S506). In this stability analysis process, it is determined whether the peak position moves according to the movement of the host vehicle. Then, the distance between the host vehicle and the road surface structure ST is calculated (S507), and a final output is made (S508). In the final output of S508, a process of outputting an alarm or an emergency brake signal to the vehicle control device for the occupant is performed, and the vehicle control device controls the alarm and the emergency brake. However, the present invention is not limited thereto. For example, only the information on the distance between the host vehicle and the road surface structure ST may be output to the vehicle control device, and the vehicle control device may determine the alarm and the emergency brake.

[0034] FIG. 6 is a flowchart for explaining the content of information processing in the server NS and the information-using vehicle C2.

[0035] The information-using vehicle C2 measures the position of its own vehicle with the self-position measuring device 325 and transmits the self-position information to the server NS (S601). When the server NS receives the provision of the self-position information from the information-using vehicle C2, it searches the server database DBs and extracts the position information of the road surface structures existing around the self-position of the information-using vehicle C2 (S602). Then, it distributes the position information to the information-using vehicle C2 that provided the self-position information.

[0036] When the information-using vehicle C2 receives the position information of the road surface structure ST existing around its own position from the server NS, it stores it in the vehicle database DBc (S603). Then, it captures an image of the front of its own vehicle with the stereo camera 322 to obtain a captured image (S604), and performs detection of the road surface structure ST by image processing using the position information of the road surface structure ST received from the server NS and stored in the vehicle database DBc (S605).

[0037] In the image processing of S605, based on the position information of the road surface structure ST received from the server NS, the image processing method is determined. For example, first, a process of searching the vehicle database DBc using the position information of the road surface structures around the self-position measured by the self-position measuring device 325 is performed. And when the road surface structure ST to be detected exists around the self-position, at least one of the image processing algorithm and parameters is changed by the image processing method determination unit of the stereo camera 322.

[0038] The stereo camera 322 performs image processing using the changed algorithm and parameters to detect the road surface structure ST. The stereo camera 322 changes the algorithm or parameters based on the position information of the road surface structure ST received from the server NS, analyzes in detail the image processing area corresponding to the position, and adjusts the detection accuracy.

[0039] The information-utilizing vehicle C2 performs vehicle control by using the detection result of the road surface structure ST by the stereo camera 322 (S606). In the vehicle control process of S606, for example, at least one of adjustment of the vehicle speed for passing through the road surface structure ST, route setting, and warning to the occupant is performed. For example, the vehicle control device 326 searches the vehicle database DBc for information on the road surface structure ST around the self-position measured by the self-position measuring device 325, and compares it with the information on the passing speed stored corresponding to the road surface structure ST. If it is determined that the speed of the host vehicle has exceeded, deceleration control is performed. Also, when the information on the road surface structure ST stored in the vehicle database DBc is a speed bump, the vehicle is decelerated to a speed at which the speed bump can be passed safely or comfortably, or the upper limit speed of the ACC for vehicle-to-vehicle distance control is changed to a speed at which it can be passed safely or comfortably. Then, the vehicle route generation device 328 performs route setting so that the information-utilizing vehicle C2 passes through a road or lane on which it can travel safely or comfortably. The human machine interface 327 sounds a warning alarm or displays a warning on the monitor to warn the occupant when it is determined that the speed of the information-utilizing vehicle C2 has exceeded. Note that when the information-utilizing vehicle C2 has an active suspension capable of controlling the damping value of the suspension, the damping value for suppressing the behavior of the vehicle when passing through a speed bump may be adjusted.

[0040] FIG. 7 is a flowchart for explaining the processing by the stereo camera of the information-utilizing vehicle C2. The description of the configuration similar to the flowchart shown in FIG. 5 is omitted.

[0041] The stereo camera 322 changes the image processing method according to whether the road surface structure ST exists around the self-position. When the road surface structure ST does not exist around the self-position, the stereo camera 322 performs normal image processing using the same algorithm and parameters as the information-providing vehicle C1. On the other hand, when the road surface structure ST exists around the self-position, at least one of the image processing algorithm and parameters is changed compared to the case where it does not exist, and image processing is performed.

[0042] When the structure ST on the road surface exists around its own position, in the processing area setting process of S503A and the parallax detail analysis process for each lateral search of S504A, based on the position information of the structure ST on the road surface received from the server NS, a process of narrowing down the image processing area of the road surface unevenness and a process of limiting the peak position are performed. FIG. 9 is a diagram for explaining a method of image processing using position information, and shows an example of narrowing down the image processing area.

[0043] For example, as shown in FIG. 9, when the information-utilizing vehicle C2 approaches the position of the structure ST on the road surface received from the server NS and is determined to have entered within a predetermined range, the image processing method determination unit of the stereo camera 322 performs a process of narrowing down the image processing area of the road surface unevenness based on the position information of the structure ST on the road surface received from the server NS. In the example shown in FIG. 9, the narrowing is performed from the first image processing areas 821 and 822 set in the depth direction on the tire traveling path to the second image processing areas 911 and 912 on the tire traveling path and corresponding to the position of the structure ST on the road surface.

[0044] Furthermore, by performing time-world coordinate conversion (conversion of the position information of an object into a three-dimensional to two-dimensional coordinate system considering time-varying information such as imaging, position measurement, and vehicle speed, and the coordinate system of the map and the imaging coordinate system by the camera) taking into account the self-position of the vehicle estimated by receiving radio waves from GPS or base stations, the update cycle of GPS information, the vehicle speed at that time, the position information of the road surface structure ST received from the server NS, the estimated error of the map at the current driving time (which also depends on radio wave conditions and the number of detected base stations), and time information such as information transmission in the in-vehicle communication environment and the imaging timing of the images captured by the camera device, it becomes possible to narrow down the area on the image captured by the camera where road surface unevenness should be analyzed to an even smaller part of the tire travel path. As a result, by partially increasing the density (the calculation target points and granularity) of the parallax image calculation S503 and the calculation of road surface unevenness S504, S505, it is possible to improve the processing accuracy while suppressing an increase in the calculation load as much as possible. Also, processing that leads to an improvement in accuracy, such as increasing the probability of correct determination by relaxing the threshold value at the time of road surface unevenness determination (for example, the height determination threshold value in peak calculation S505), becomes possible.

[0045] Then, the image processing within the second image processing regions 911, 912 is changed to higher-precision image processing than the image processing in the first image processing region, and the road surface structure ST is detected. The second image processing regions 911, 912 are set to have a shorter length in the depth direction than the first image processing regions 821, 822, and have a size and shape that cover the road surface structure ST in the parallax image. In this way, by narrowing down and reducing the image processing region, the load of image processing can be reduced. Therefore, within limited hardware resources, more detailed image processing can be performed in a short time, and the road surface structure ST, which is the detection target object, can be detected with high precision.

[0046] FIG. 11 is a diagram showing the result of peak detection of road surface unevenness, and shows the detected value of the road surface height in the depth direction of the tire travel path. In the graph of FIG. 11, the horizontal axis is the distance (m) in the depth direction, and the vertical axis is the road surface height (cm) detected by the stereo camera 322.

[0047] When the image processing method determination unit of the stereo camera 322 determines that the structure ST on the road surface does not exist around the self-position, it determines the peak position of the road surface unevenness within the first detection range set along the depth direction of the tire travel path. When the structure ST on the road surface exists around the self-position, it changes the algorithm or parameters of the image processing by the image processing unit, and determines the peak position of the road surface unevenness within the second detection range on the tire travel path and corresponding to the position of the structure ST on the road surface (S505A).

[0048] Also, the image processing method determination unit of the stereo camera 322 sets the determination threshold value for determining the peak position of the road surface unevenness within the second detection range to a value larger than the determination threshold value for determining the peak position of the road surface unevenness within the first detection range (S505A). Thereby, noise before and after the peak position can be removed, the computational load for local peak detection of the road surface unevenness can be reduced, and the determination accuracy of the structure ST on the road surface can be improved.

[0049] In the final output of S508A, similar to S508, processing for outputting signals such as warnings and emergency brakes to the vehicle control device, or processing for outputting only information on the distance between the host vehicle and the structure ST on the road surface is performed. Then, further, it is determined whether the reliability of the captured image captured by the stereo camera 322 is equal to or greater than the threshold value. For example, when the reliability is lower than the threshold value due to weather or external light conditions, the processing in S501 to S507 is skipped, and the final output is made based on the position information of the structure ST on the road surface distributed from the server NS, and vehicle control such as warnings to the occupants, suspension pressure, deceleration, and steering is performed based on the final output.

[0050] Also, in this embodiment, after step S508A, a step S509A for calculating verification information of the structure ST on the road surface is provided.

[0051] In the verification information calculation process of S509A, the stereo camera 322 performs a process of converting the position of the road surface structure ST detected by image processing into the coordinate position in the world coordinate system using the self-position information obtained by the self-position measurement device 325, and a process of acquiring information on the position where the behavior of the host vehicle when passing over the road surface structure ST is detected.

[0052] The process of converting the position of the road surface structure ST into world coordinates is performed by considering (1) the vehicle speed of the host vehicle at the time of imaging, (2) the imaging timing by the imaging unit of the stereo camera 322, (3) the self-position information obtained by the self-position measurement device 325, (4) the acquisition timing of the self-position information by the self-position measurement device 325, (5) the communication delay speed of in-vehicle devices, etc. from the distance to the road surface structure ST detected by image processing. The time-world coordinate conversion process is to perform the conversion of position information into a three-dimensional - two-dimensional coordinate system considering the time-varying information such as imaging, position measurement, and vehicle speed, and the coordinate system of the map and the imaging coordinate system by the camera. In addition, the detection of the behavior of the host vehicle when passing over the road surface structure ST may detect changes in pitching, vertical acceleration, and suspension pressure by in-vehicle sensors such as a G-sensor in addition to detecting the vertical movement of the captured image by the stereo camera 322.

[0053] In the verification information calculation of S509A, a process is performed to verify the position of the road surface structure ST detected by the stereo camera 322 of the information-using vehicle C2 from the behavior of the host vehicle and transmit the verification information to the server NS. The server NS can improve the estimation accuracy of the position of the road surface structure ST using the verification information.

[0054] <Second Embodiment> Next, the external recognition system in the second embodiment will be described with reference to the drawings. FIG. 13 is a flowchart for explaining the processing content of the external recognition system in the second embodiment, and FIG. 14 is a diagram showing an example of the situation where the processing shown in FIG. 13 is performed.

[0055] What is characteristic in this embodiment is that, although the stereo camera 312 of the information providing vehicle C1 could not detect the road surface structure ST, the server NS performs image processing using an image obtained by imaging the vibration detection position because the vehicle detected vertical vehicle vibrations caused by passing over the road surface structure ST.

[0056] As shown in FIG. 14, for example, the stereo camera 312 of the information providing vehicle C1 acquires captured images captured during traveling and stores them in the vehicle database (S1301). Then, when vibrations equal to or greater than a threshold value preset to detect vertical vehicle vibrations caused by passing over the road surface structure ST are detected by, for example, the G sensor 313 (S1302), position information of the vibration detection point Pn is acquired based on the self-position information by the self-position measuring device (S1303). The method for detecting vehicle vibrations is not limited to the G sensor 313. For example, the up-and-down movement of the captured image by the stereo camera 312, pitching by in-vehicle sensors such as a gyro sensor, up-and-down acceleration, and changes in suspension pressure may be used.

[0057] The stereo camera 312 searches the vehicle database and extracts the captured image in which the vibration detection point Pn was imaged from the captured images stored in the vehicle database (S1304). Here, as shown in FIG. 14, the captured image captured when going back t hours from the time when the vertical vehicle vibrations were detected to the position Pn-1 where the stereo camera 312 imaged the vibration detection point Pn is extracted as the image in which the vibration detection point Pn is imaged. The road-vehicle communication device 317 receives information on the vibration detection point Pn and the captured image in which the point was imaged from the stereo camera 312 and transmits it to the server NS.

[0058] When the server NS acquires the information of the vibration detection point Pn and the captured image of the vibration detection point Pn from the information-providing vehicle C1, the data analysis device 303 performs image processing to detect the road surface structure ST from the captured image of the vibration detection point Pn (S1305). The image processing performed by the server NS is different from the image processing performed by the stereo camera 312 in terms of processing content, and has a more detailed and advanced processing content than the image processing performed by the stereo camera 312. Since the server NS has no processing time constraints like the stereo camera 312 and has higher-spec and richer hardware resources and software resources than the stereo camera 312, it can perform more detailed and advanced image processing. Therefore, the road surface structure ST that could not be detected by the stereo camera 312 can be detected, and the reason why it could not be detected by the stereo camera 312 can be verified.

[0059] The server NS stores the detection result of the road surface structure ST by the image processing of the data analysis device 303 and the position information of the vibration detection point Pn in the server database DBs (S1306), analyzes the position information of the road surface structure ST at a predetermined timing, and recalculates the position information (S1307).

[0060] According to the external environment recognition system of the present embodiment, the road surface structure ST that could not be detected by the stereo camera 312 can be detected by the image processing in the server NS, and the accuracy of the position information of the road surface structure ST stored in the server database DBs can be made higher.

[0061] In addition, in the present embodiment, although it has been described that when the on-road structure ST cannot be detected by the stereo camera 312 of the information-providing vehicle C1, image processing for verification is performed in the server NS, even when the on-road structure ST is detected by the stereo camera 312, image processing for verification may be performed in the server NS. Further, in the present embodiment, the case of the information-providing vehicle C1 has been described as an example, but in the information-using vehicle C2, it is also possible to configure such that vertical vehicle vibrations are detected by a G-sensor 323 or the like, and image processing is performed in the server NS using the captured image at the point where the vehicle vibrations are detected.

[0062] <Third Embodiment> Next, the external recognition system in the third embodiment will be described with reference to the drawings. FIG. 15 is a flowchart for explaining the processing contents of the external recognition system in the third embodiment, and shows an analysis method based on the position of the on-road structure and the information of vehicle vibrations.

[0063] What is characteristic in the present embodiment is that in addition to the position information of the on-road structure ST, information on the degree of influence on vehicle vibrations when the vehicle actually passes is stored in the server, and is used for vehicle control when the information-using vehicle C2 passes over the on-road structure ST.

[0064] The information-providing vehicle C1 acquires a captured image (S1501), and by image processing, detects the on-road structure ST, and measures the distance to the on-road structure ST and the height of the on-road structure ST from the road surface reference plane (S1502). Note that the road surface reference plane can be obtained, for example, from the average position of the road surface unevenness in the depth direction. Then, the information-providing vehicle C1 detects the vibration when passing over the on-road structure ST by the G-sensor 323 (S1503), and acquires the position information of the vibration detection point (S1504).

[0065] Then, based on the magnitude of the vibration detected by the G-sensor 323, the degree of influence on vehicle vibration is calculated, and the position information of the road surface structure ST, the position of the vibration detection point, and the information on the degree of influence on vehicle vibration are transmitted to the server NS. The server NS stores these pieces of information in the server database DBs (S1505). The server NS aggregates the position information of the road surface structure ST stored in the server database DBs, the position of the vibration detection point, and the information on the degree of influence on vehicle vibration, statistically analyzes the big data using the data analysis device 303, updates the driving experience information, maps it onto the road-related map, and creates traffic experience information on the map. When the server NS stores the position information of the road surface structure ST in the server database DBs, it recalculates the traffic experience information on the map at a predetermined timing to update it with the latest information (S1506).

[0066] When the server NS receives the provision of its own position information from the information-providing vehicle C1, it transmits the position information of the road surface structure ST existing around the own position of the information-using vehicle C2 and the information on the degree of influence on vehicle vibration to the information-using vehicle C2.

[0067] Based on the position information of the road surface structure ST existing around its own position and the information on the degree of influence on vehicle vibration, the information-using vehicle C2 performs vehicle control such as decelerating the traveling speed and adjusting the damping force of the suspension so as to reduce the vibration of the vehicle.

[0068] According to the present embodiment, since the information on the degree of influence of the road surface structure ST on vehicle vibration can be provided to the information-using vehicle C2, in the information-using vehicle C2, vehicle control can be actively performed so as to reduce vibration, and the riding comfort of the passengers can be improved.

[0069] As described above, the embodiments of the present invention have been described in detail. However, the present invention is not limited to the above-described embodiments, and various design changes can be made without departing from the spirit of the present invention described in the claims. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Also, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Furthermore, for a part of the configuration of each embodiment, addition, deletion, or replacement with other configurations is possible.

Explanation of Signs

[0070] C1 Information-providing vehicle C2 Information-utilizing vehicle (own vehicle) NS Server (external server) DBs Server database DBc Vehicle database ST Structure on road surface 322 Stereo camera 323 G sensor 324 Road-vehicle communication device (vehicle communication unit) 325 Self-position measurement device 821, 822 First image processing area 911, 912 Second image processing area

Claims

1. An external recognition system comprising at least one or more information-providing vehicles that are communicatively connected to an external server and transmit information, wherein the information-providing vehicle, has a road surface structure detection unit that measures the three-dimensional shape of the driving road surface and detects structures on the road surface, the information-providing vehicle includes a vehicle vibration detection unit that detects vehicle vibration in the vertical direction and a stereo camera as the road surface structure detection unit, when the stereo camera detects vehicle vibration equal to or greater than a threshold value by the vehicle vibration detection unit, the stereo camera extracts a captured image in which the vibration detection point where the vehicle vibration is detected is imaged, and transmits the captured image in which the vibration detection point is imaged to the external server, the external server performs image processing to detect the road surface structure from the captured image in which the vibration detection point acquired from the host vehicle is imaged. An external recognition system characterized by this.

2. In the external recognition system according to claim 1, the image processing performed by the external server is processing with higher accuracy than the image processing performed by the information-providing vehicle.

Images