Vehicle control system and computer program

Abstract

The present invention provides a vehicle control device and computer program capable of suppressing misidentification of pedestrian crossings when detecting road markings. [Solution] The vehicle control device 10 includes a control unit 11 that performs driving support control of the vehicle 1. The control unit analyzes image data captured from the environment around the vehicle, recognizes roads in the direction the vehicle is traveling in the environment, calculates a first distance in the width direction of the road, recognizes road markings on the road, and if it recognizes that the road markings are a predetermined pattern indicating a pedestrian crossing, it calculates a second distance in the width direction of a predetermined area including the road markings, calculates the ratio of the second distance to the first distance, and if the ratio is greater than or equal to a predetermined value, it recognizes the road markings as a pedestrian crossing and performs driving support control. If the ratio is less than the predetermined value, it recognizes the road markings as not being a pedestrian crossing and does not perform driving support control.

Description

Technical Field

[0001] The present invention relates to a vehicle control device and a computer program for performing driving assistance.

Background Art

[0002] For example, Patent Document 1 describes a technique for recognizing a crosswalk painted on a road based on imaging data. The technique described in Patent Document 1 is configured to recognize a white line existing on a road based on imaging data obtained by imaging the environment in front of a vehicle, and when a white line having a predetermined width repeatedly appears, recognize it as a crosswalk.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Characters are painted on the road in various patterns. According to the technique described in Patent Document 1, there is a possibility of misrecognizing characters painted on the road as a crosswalk.

[0005] An object of the present invention is to provide a vehicle control device and a computer program capable of suppressing misrecognition of a crosswalk when detecting a road surface marking.

Means for Solving the Problems

[0006] One aspect of the present invention is a vehicle control device comprising a control unit that performs driving assistance control of a vehicle, wherein the control unit analyzes image data obtained by imaging the environment around the vehicle, recognizes a road in the direction the vehicle is traveling in the environment, calculates a first distance in the width direction of the road, recognizes road markings present on the road, and if the road markings are recognized as a predetermined pattern indicating a pedestrian crossing, calculates a second distance in the width direction of a predetermined area including the road markings, calculates the ratio of the second distance to the first distance, recognizes the road markings as a pedestrian crossing and performs the driving assistance control if the ratio is greater than or equal to a predetermined value, and recognizes the road markings as not being a pedestrian crossing and does not perform the driving assistance control. [Effects of the Invention]

[0007] According to the present invention, it is possible to suppress the misrecognition of pedestrian crossings when detecting road markings. [Brief explanation of the drawing]

[0008] [Figure 1] Block diagram showing the configuration of a vehicle according to the embodiment. [Figure 2] This figure shows an example of road imaging data, including road markings. [Figure 3] This figure shows an example of a method for recognizing road markings. [Figure 4] This figure shows an example of a method for measuring the relative distance between a vehicle and road markings. [Figure 5] This figure shows an example of a method for measuring road width. [Figure 6] This is a flowchart showing the processing flow of the driver assistance control method. [Modes for carrying out the invention]

[0009] As shown in Figure 1, Vehicle 1 is configured to perform driver assistance. Vehicle 1 is configured with a detection unit 2 and a vehicle control device 10 to perform driver assistance such as ADAS (Advanced Driver-Assistance Systems). The detection unit 2 detects the environment around Vehicle 1. The detection unit 2 is composed of multiple devices depending on the application. The detected values ​​detected by the detection unit 2 are used for driver assistance and navigation devices, etc.

[0010] The detection unit 2 is equipped with a camera 2A that captures images of the environment around the vehicle 1. The camera 2A captures images of the area around the vehicle 1 and outputs the captured data. In this embodiment, the camera 2A captures a predetermined range in front of the vehicle 1. The captured data from the camera 2A is used, for example, for driver assistance or as a drive recorder for the vehicle 1. The imaging range and imaging direction of the camera 2A may differ depending on the vehicle 1.

[0011] The detection unit 2 includes a LiDAR (light detection and ranging) device 2B that detects three-dimensional data around the vehicle 1. The LiDAR device 2B periodically irradiates laser light in front of or around the vehicle 1 within the detection area and measures the reflected light from the target object. The LiDAR device 2B is configured to allow adjustment of the detection area. The LiDAR device 2B scans the laser light within the detection area and acquires measurement data. The LiDAR device 2B is configured to generate three-dimensional point cloud data around the vehicle 1 based on the measurement data.

[0012] The measurements from the LiDAR device 2B are used to detect other vehicles, pedestrians, bicycles, motorcycles, and other traffic participants around vehicle 1, as well as objects around vehicle 1. The LiDAR device 2B also detects road structures present in the road environment.

[0013] The detection unit 2 includes, for example, a radar device 2C that scans radar waves to detect objects present around the vehicle 1. The radar device 2C is configured to complement the detection of objects with the lidar device 2B. The radar device 2C detects the relative distance to an object by irradiating millimeter-wave radar waves in the detection area and receiving the reflected waves reflected from the object. The radar device 2C is configured to allow adjustment of the detection area.

[0014] The measurements from radar device 2C are used to detect other vehicles, pedestrians, bicycles, motorcycles, and other traffic participants around vehicle 1, as well as objects around vehicle 1. Radar device 2C also detects road structures present in the road environment. The object recognition unit is composed of lidar device 2B and / or radar device 2C.

[0015] The detection unit 2 is equipped with a position sensor 2D that detects the current position of the vehicle 1. The position sensor 2D is, for example, a GPS (Global Positioning System) sensor or a GNSS (Global Navigation Satellite System) sensor. The position sensor 2D may be complemented by autonomous sensors (not shown) used for autonomous navigation, such as a gyro sensor and an accelerometer, to determine the position of the vehicle 1.

[0016] Vehicle 1 is equipped with an input / output unit 3 that accepts user operations and can display information. The input / output unit 3 is composed of, for example, a touch panel that accepts touch operations and can display images. The input / output unit 3 may also be composed of an input unit that accepts input operations such as physical switches and a display unit that can display information such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display.

[0017] The input / output unit 3 may include a speaker that outputs voice information, an indicator that outputs light emission information, a vibrator that outputs vibration information, and the like. The input / output unit 3 has a function as an output unit that outputs information in at least one mode such as characters, images, voices, light emission, vibration, and the like.

[0018] The vehicle 1 includes a communication unit 4 that can be communicatively connected to the network W. The communication unit 4 is a communication interface configured to enable wireless communication. The communication unit 4 communicatively connects with, for example, a wireless base station existing around the vehicle 1 and mutually communicates with various communication targets via the network W. The vehicle 1 communicates with, for example, a server device 20 communicatively connected to the network W and acquires map information including the current position of the vehicle 1.

[0019] The vehicle 1 includes a drive unit 5 that generates power for running. The drive unit 5 is configured by, for example, an internal combustion engine using fuel. The drive unit 5 is configured by an electric motor when the vehicle 1 is an electric vehicle. The drive unit 5 may be configured by combining an internal combustion engine and an electric motor when the vehicle 1 is a hybrid vehicle. The drive unit 5 is controlled by the vehicle control device 10 during the execution of driving support, and the speed is adjusted.

[0020] The vehicle 1 includes a braking unit 6 for decelerating the vehicle speed and controlling it to a stop state. The braking unit 6 is configured by, for example, a brake device that generates braking force. The braking unit 6 may be integrated with the drive unit 5 when the vehicle 1 is an electric vehicle. The braking unit 6 is controlled by the vehicle control device 10 during the execution of driving support.

[0021] The vehicle 1 includes a steering unit 7 for operating the traveling direction. The steering unit 7 is configured by a power steering device or the like that gives a steering angle to the steering wheel according to the steering operation. The steering unit 7 may be integrated with the drive unit 5 that variably controls the driving force on the left and right of the drive wheels when the vehicle 1 is an electric vehicle. The steering unit 7 is controlled by the vehicle control device 10 during the execution of driving support, and the steering angle is adjusted.

[0022] The vehicle control device 10 includes a control unit 11 that performs control related to the driving of the vehicle 1. Based on the detection values ​​detected by the detection unit 2, the control unit 11 integrates and executes control of the vehicle 1, such as driving, driving assistance, navigation, and communication via the network W. The control unit 11 is composed of at least one hardware processor such as a CPU (Central Processing Unit). The control unit 11 may be implemented by hardware (including circuitry) such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit), or it may be implemented by the cooperation of software and hardware.

[0023] The vehicle control device 10 includes a storage unit 12 for storing data and programs. The storage unit 12 is composed of a non-temporary storage medium such as a hard disk drive (HDD) or a solid-state disk (SSD). The storage unit 12 stores computer programs and data necessary for controlling the vehicle 1. The programs may be stored in the storage unit 12 in advance, or they may be stored on an externally connectable storage medium such as a DVD or CD-ROM and installed in the storage unit 12 when the storage medium is mounted on the drive device. The control unit 11 controls the drive unit 5, braking unit 6, and steering unit 7 based on the detection values ​​detected by the detection unit 2, and performs driving assistance control such as Obstacle Anticipation Assist (OAA).

[0024] As shown in Figure 2, Vehicle 1 is configured to perform driving assistance control based on the recognition results of image data M1, which captures the environment around Vehicle 1. The control unit 11 acquires the image data M1 captured by camera 2A. The control unit 11 has previously performed machine learning such as deep learning using image data of the road environment being driven on as training data, and is configured to recognize the environment around Vehicle 1, as well as objects, signs, road markings, etc., present in the environment. The control unit 11 extracts objects included in the image data M1 using image recognition technology such as semantic segmentation.

[0025] The control unit 11 classifies the pixels of the image data capturing the road environment being driven on and divides the image data into regions. The control unit 11 labels the divided regions and recognizes objects. The control unit 11 is configured to calculate the distance of an object in the depth direction from the vehicle 1 based on the horizontal (x direction) coordinates and vertical (y direction) coordinates in the image data. The control unit 11 calculates the distance between the object and the vehicle 1 in the image data and also calculates the dimensions of the object based on the object's coordinates in the image.

[0026] The control unit 11 analyzes the imaging data M1, which captures the environment around the vehicle 1. The control unit 11 recognizes the road R that exists in the direction the vehicle is traveling from within the environment around the vehicle 1. The control unit 11 recognizes structures related to road R, such as road structures, road signs, road markings P, buildings B, and other traffic participants such as other vehicles and pedestrians. For example, the control unit 11 recognizes the road R that exists in the direction G that the vehicle 1 is traveling in within the environment included in the imaging data M1.

[0027] The control unit 11 recognizes road R by setting the road boundary line Ln (n: a natural number) as the road boundary line Ln if such a line exists on the road R. The control unit 11 recognizes the road surface between a pair of adjacent road boundary lines Ln as road R. The control unit 11 recognizes the lane boundary line C1 existing between a pair of road boundary lines Ln as the center line or lane boundary line of road R. In the illustrated example, the control unit 11 recognizes the road surface between road boundary line L1 and road boundary line L2 as road R, and recognizes the lane boundary line C1 existing between road boundary line L1 and road boundary line L2 as the center line or lane boundary line of road R. The road boundary line Ln and lane boundary line C1 are indicated by, for example, white or yellow continuous lines, dashed lines, etc.

[0028] If the control unit 11 cannot recognize a boundary line on road R, it sets a virtual road boundary based on the arrangement of structures adjacent to the road. If no road boundary line Ln exists on road R, the control unit 11 extracts characteristic features of structures present in the road environment, such as the color of the pavement, the presence or absence of sidewalks, the presence or absence of road structures such as guardrails, and the positional relationship with building B, and sets a virtual road boundary line LMm (m: natural number) on road R.

[0029] The control unit 11 performs a process to treat the set virtual road boundary line LMm as a road boundary. The control unit 11 recognizes the road surface between the pair of virtual road boundary lines LMm as road R. The control unit 11 may also recognize the road surface between the road boundary line Ln and the virtual road boundary line LMm as road R. Alternatively, the control unit 11 may recognize road R by comparing the road map information included in the map information acquired from the server device 20 with the position data of the position sensor 2D.

[0030] When the control unit 11 extracts a road marking P present on the road R, it determines whether the road marking P is a pedestrian crossing. The control unit 11 performs a first determination process to recognize a pedestrian crossing from the image data M1, for example, using an image recognition method such as pattern matching. In the first determination process, the control unit 11 compares the extracted image of the road marking P with a template image of a pedestrian crossing. The database of template images is stored in, for example, the storage unit 12. The template image shows a predetermined stripe-like pattern that is characteristic of a pedestrian crossing. The database of template images includes, for example, multiple images of pedestrian crossings captured by an in-vehicle camera.

[0031] The control unit 11 compares the extracted road marking P image with a template image and calculates the similarity. If the road marking P extracted from the imaging data M1 has a high similarity to a predetermined pattern indicating a pedestrian crossing, the control unit 11 recognizes the road marking P as a candidate for a pedestrian crossing. The control unit 11 sets a predetermined region Q that includes the road marking P, for example.

[0032] As shown in Figure 3, the imaging data M2 of road R may contain not only the first road marking P1 indicating a pedestrian crossing, but also other second road markings P2 indicating text information, etc. A portion of the second road markings P2 contains one or more predetermined regions Qr (r: natural number) that are similar to a predetermined pattern indicating a pedestrian crossing. In this case, the control unit 11 may recognize the second road markings P2 as a candidate for a pedestrian crossing in the first determination process. In addition to the first determination process, the control unit 11 performs a second determination process to improve the recognition accuracy of the road markings P.

[0033] In the first determination process, if the control unit 11 recognizes that the first road marking P1 and the second road marking P2 form a predetermined pattern indicating a pedestrian crossing, it calculates a first distance D1 in the width direction of road R in the second determination process. The width direction of road R is the short direction of road R which is substantially perpendicular to the longitudinal direction of road R. The first distance D1 includes one or more lanes in the direction G from which the vehicle 1 is traveling, and one or more opposing lanes. The control unit 11 calculates the first distance D1 in the vicinity of the first road marking P1 and the second road marking P2.

[0034] The control unit 11 calculates a second distance in the width direction of a predetermined area including road markings. The second distance may include one or more lanes in the direction G on which the vehicle 1 travels, and one or more opposing lanes. For example, the control unit 11 sets a predetermined area QA including a first road marking P1 of a predetermined pattern. The control unit 11 sets a predetermined area Qr including a second road marking P2 of a predetermined pattern. The control unit 11 calculates a second distance DA1 in the width direction of the predetermined area QA. The control unit 11 calculates a second distance DBr in the width direction of the predetermined area Qr. The control unit 11 calculates the ratio of the calculated second distance to the first distance D1 based on the following formula (1). Ratio = 2nd distance / 1st distance (1)

[0035] Since the predetermined area QA is a pedestrian crossing, the second distance DA1 is approximately equal to the first distance D1, so the ratio is close to 1. If the predetermined area Qr is not a pedestrian crossing, the second distance DBr is shorter than the first distance D1, so the ratio is less than 1. Since the second distance DBr is a road marking such as letters, it is highly likely to be less than half of the first distance D1. The control unit 11 compares the calculated ratio with a preset threshold. The threshold is set to a value less than 1, such as 0.5.

[0036] The control unit 11 recognizes the road marking P as a pedestrian crossing if the calculated ratio is above a threshold (above a predetermined threshold). When the control unit 11 recognizes a pedestrian crossing, it performs driving support control for the pedestrian crossing. When the pedestrian crossing approaches the vehicle 1 within a predetermined distance, the control unit 11 controls the drive unit 5, braking unit 6, and steering unit 7 to perform driving support control for the pedestrian crossing, such as deceleration, slowing down, stopping, and avoidance maneuvers. When the calculated ratio is below a threshold (below a predetermined threshold), the control unit 11 recognizes the second road marking P2 as not being a pedestrian crossing. When the control unit 11 recognizes the second road marking P2 as not being a pedestrian crossing, it does not perform driving support control.

[0037] As shown in Figure 4, when the control unit 11 performs driving assistance control for a pedestrian crossing, it calculates the relative distance from the vehicle 1 to the first road marking P1. If map information is available, the control unit 11 obtains map information including the current location of the vehicle 1 from the server device 20 and / or storage unit 12. The control unit 11 extracts road map information related to road R from the map information and calculates the relative distance (third distance DA2) from the vehicle 1 to the first road marking P1 based on the first position information of the road marking P included in the road map information and the second position information of the current location. The control unit 11 may also calculate the relative distance from the vehicle 1 to the first road marking P1 based on the vertical (Y direction) coordinates in the image data M3 in which the pedestrian crossing is imaged.

[0038] On road R, the depth dimension DA3 of the pedestrian crossing is set to a predetermined value. When the control unit 11 recognizes the first road marking P1 as a pedestrian crossing, it calculates the depth dimension DA3 of a predetermined area QA of the pedestrian crossing in the image data M3. Based on the comparison result of the calculated dimension DA3 with the predetermined value, the control unit 11 may calculate the third distance DA2 from the vehicle 1 to the first road marking P1. Based on the relative distance and the vehicle speed, the control unit 11 calculates the time to reach the pedestrian crossing. For example, if the time to reach the pedestrian crossing is less than or equal to a preset predetermined value, the control unit 11 executes driving support control corresponding to the pedestrian crossing.

[0039] Figure 5 shows an example of a method for calculating the first distance D1 in the width direction of road R based on imaging data M4. The control unit 11 recognizes, for example, a first boundary and a second boundary that indicate the boundary of road R in the calculation of the first distance D1. In the illustrated example, the control unit 11 recognizes the road boundary line L1 and / or virtual road boundary line LM1 as the first boundary. The control unit 11 recognizes the road boundary line L2 and / or virtual road boundary line LM2 as the second boundary. The control unit 11 recognizes road R based on the first boundary and the second boundary. The control unit 11 recognizes one or more lane boundary lines C1 on road R.

[0040] The control unit 11 recognizes the lanes on the road R based on the lane boundary line C1. The control unit 11 recognizes the driving lane R1 on which the vehicle 1 is traveling among the recognized lanes. The control unit 11 sets a virtual trajectory V1 that extends the position of the vehicle 1 in the driving lane R1 in the direction of travel. The virtual trajectory V1 indicates the position of the vehicle 1 in the driving lane R1. The control unit 11 calculates a fourth distance DA4 in the width direction relative between the first boundary and the virtual trajectory V1 indicating the position of the vehicle 1. The control unit 11 calculates a fifth distance DA5 in the width direction relative between the second boundary and the virtual trajectory V1 indicating the position of the vehicle 1. The control unit 11 calculates a first distance D1 by adding the fourth distance DA4 and the fifth distance DA5, for example.

[0041] Figure 6 shows the processing flow of the driver assistance control method executed in the vehicle control device 10. The driver assistance control method is executed based on a computer program installed in the computer mounted on the vehicle control device 10. The computer program causes the vehicle control device 10 to perform the following processes.

[0042] The control unit 11 acquires the image data captured by the camera 2A (S100). The control unit 11 analyzes the image data capturing the environment around the vehicle (S102). The control unit 11 recognizes the road R that exists in the direction the vehicle is traveling in the environment (S104). The control unit 11 determines whether or not there is a road marking P on the road R (S106). If the control unit 11 does not recognize a road marking P, it returns to S100 and continues processing. If the control unit 11 recognizes a road marking P, it determines whether or not the road marking P is a predetermined pattern indicating a pedestrian crossing (S108).

[0043] If the road marking P is not a predetermined pattern, the control unit 11 returns to processing S100 and continues processing. If the road marking P is a predetermined pattern, the control unit 11 calculates a first distance D1 in the width direction of the road R (S110). If the control unit 11 recognizes that the road marking P is a predetermined pattern indicating a pedestrian crossing, it calculates a second distance in the width direction of a predetermined area Q including the road marking P (S112). The control unit 11 calculates the ratio of the second distance to the first distance and determines whether the ratio is greater than or equal to a predetermined value (S114).

[0044] If the ratio is less than a predetermined value, the control unit 11 determines that the road marking P is not a predetermined pattern and returns to S100 without performing any driving assistance. If the ratio is greater than or equal to a predetermined value, the control unit 11 recognizes the road marking P as a pedestrian crossing (S116). The control unit 11 then performs driving assistance control for the pedestrian crossing (S118).

[0045] As described above, the vehicle control device 10 can suppress the misrecognition of a pedestrian crossing when detecting a road marking P. The vehicle control device 10 can prevent the misrecognition of the road marking P as a pedestrian crossing by executing a second determination process, even if the first determination process determines that the road marking P is a predetermined pattern. The vehicle control device 10 can suppress the execution of driver assistance control due to misrecognition of a pedestrian crossing.

[0046] In the embodiments described above, the computer programs executed in each configuration of the vehicle control device 10 may be provided in the form of being recorded on a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. The computer programs may also be provided as a program product. [Explanation of Symbols]

[0047] 1 Vehicle, 2 Detection unit, 2A Camera, 2B LiDAR device, 2C Radar device, 2D Position sensor, 3 Input / Output unit, 4 Communication unit, 5 Drive unit, 6 Braking unit, 7 Steering unit, 10 Vehicle control device, 11 Control unit, 12 Storage unit, 20 Server device, B Building, C1 Lane boundary line, D1 First distance, DA1, DBr Second distance, DA2 Third distance, DA3 Dimensions, DA4 Fourth distance, DA5 Fifth distance, G Direction, LMm Virtual road boundary line, Ln Road boundary line, M1, M2, M3, M4 Image data, P Road marking, P1 First road marking, P2 Second road marking, Q Predetermined area, QA Predetermined area, Qr Predetermined area, R Road, R1 Driving lane, W Network

Claims

1. It includes a control unit that performs driver assistance control for the vehicle, The control unit, By analyzing the image data captured from the environment surrounding the vehicle, In the aforementioned environment, the road present in the direction of travel of the vehicle is recognized. The first distance in the width direction of the aforementioned road is calculated, Recognizing the road markings present on the aforementioned road, If the road marking is recognized as a predetermined pattern indicating a pedestrian crossing, a second distance in the width direction of the predetermined area including the road marking is calculated. The ratio of the second distance to the first distance is calculated, If the ratio is greater than or equal to a predetermined value, the road marking is recognized as the pedestrian crossing, and the driver assistance control is executed. If the ratio is less than a predetermined value, the road marking is recognized as not being a pedestrian crossing, and the driver assistance control is not performed. Vehicle control system.

2. The control unit, When the aforementioned road marking is recognized as the aforementioned pedestrian crossing, the dimensions and coordinates in the depth direction of the predetermined area in the imaging data are calculated. Based on the aforementioned dimensions and coordinates, the relative third distance from the vehicle to the pedestrian crossing is calculated. Based on the third distance and the vehicle's speed, the time to reach the pedestrian crossing is calculated. If the arrival time is less than or equal to a predetermined value, the driver assistance control corresponding to the pedestrian crossing is executed. The vehicle control device according to claim 1.

3. The control unit, Recognizing the first boundary and the second boundary that indicate the boundary of the aforementioned road, The fourth relative distance in the width direction between the first boundary and the vehicle is calculated. The relative fifth distance in the width direction between the second boundary and the vehicle is calculated, Based on the fourth distance and the fifth distance, the first distance is calculated. The vehicle control device according to claim 1.

4. The control unit, The vehicle acquires map information including its current location. From the aforementioned map information, road map information related to the aforementioned road is extracted, Based on the first location information of the pedestrian crossing included in the road map information and the second location information of the current location, the relative distance from the vehicle to the pedestrian crossing is calculated. The vehicle control device according to claim 1.

5. A computer program installed in a vehicle control system that performs driver assistance control for a vehicle, By analyzing the image data captured from the environment surrounding the vehicle, In the aforementioned environment, the road present in the direction of travel of the vehicle is recognized. The first distance in the width direction of the aforementioned road is calculated, Recognizing the road markings present on the aforementioned road, If the road marking is recognized as a predetermined pattern indicating a pedestrian crossing, a second distance in the width direction of the predetermined area including the road marking is calculated. The ratio of the second distance to the first distance is calculated, If the ratio is greater than or equal to a predetermined value, the driving support control is executed. If the ratio is less than a predetermined value, the driver assistance control is not executed, or the computer is made to execute the process. Computer program.

Images