Vehicle control device, vehicle control method, and control program

Abstract

The present invention provides a vehicle control device that can accurately identify a reducing lane from a lane reduction sign. [Solution] The vehicle control device 1 includes a sign recognition unit 60 that recognizes a sign portion 80 corresponding to a lane reduction sign 50, which includes one or two folded line portions 51 including a bent portion 51A and a plurality of straight line portions 52 that are straight solid or dashed lines, from an image captured by a camera 13 that captures the area around the vehicle 2, and a reduction lane identification unit 61 that obtains one or two first shapes 81A, 83A corresponding to one or two folded line portions 51 and a plurality of second shapes 81B, 83B corresponding to the plurality of straight line portions 52 from the sign portion 80, and identifies the position of the reduction lane based on one or two first shapes and a plurality of second shapes.

Description

Technical Field

[0001] The present invention relates to a vehicle control device, a vehicle control method, and a control program.

Background Art

[0002] In recent years, efforts to provide a sustainable transportation system that takes into account people in vulnerable positions among traffic participants have been active. In order to further improve traffic safety and convenience toward this realization, research and development on driving support technologies have been conducted.

[0003] Patent Document 1 discloses a vehicle control device having an imaging unit that acquires a front image of a vehicle, a decreasing lane specifying unit that detects a lane number decrease sign from the front image and specifies a decreasing lane from the lane number decrease sign, and a travel control unit that executes steering assist control to another lane when the vehicle is present in the decreasing lane. The decreasing lane specifying unit specifies the number of lanes indicated by a road sign or a construction signboard and the position of the decreasing lane by performing pattern matching processing by referring to templates of road signs and construction signboards stored in advance in a storage unit. The templates for performing the pattern matching processing are created based on lane number decrease signs that may exist in Japan and foreign countries.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, as shown in Figure 9, there are various types of lane reduction signs, and it is difficult to memorize all of them as templates. Therefore, when encountering an unfamiliar lane reduction sign, the accuracy of identifying the reducing lane may decrease. Also, because the appearance of the lane reduction sign in the image changes depending on the angle of the lane reduction sign relative to the camera, the accuracy of identifying the reducing lane may decrease even with a known lane reduction sign.

[0006] One aspect of the present invention, in view of the above background, aims to provide a vehicle control device, a vehicle control method, and a control program that can accurately identify a lane reduction sign. [Means for solving the problem]

[0007] To solve the above problems, one aspect of the present invention provides a vehicle control device comprising: a sign recognition unit that recognizes a sign portion corresponding to a lane reduction sign, which includes one or two folded line portions including a bent portion and a plurality of straight lines that are straight solid or dashed lines, from an image captured by a camera that captures the area around a vehicle; and a reduction lane identification unit that obtains one or two first shapes corresponding to one or two of the folded line portions and a plurality of second shapes corresponding to the plurality of straight lines from the sign portion, and identifies the position of the reduction lane based on the one or two first shapes and the plurality of second shapes.

[0008] Another aspect of the present invention is a vehicle control method performed by a computer, in which a forward image captured by a camera that images the area around the vehicle recognizes a sign portion corresponding to a lane reduction sign, which includes one or two folded line portions including a bend and a plurality of straight lines which are straight solid or dashed lines, and from the sign portion, obtains one or two first shapes corresponding to one or two of the folded line portions and a plurality of second shapes corresponding to the plurality of straight lines, and determines the position of the reducing lane based on the one or two first shapes and the plurality of second shapes.

[0009] Another aspect of the present invention is a control program for causing a computer to execute a vehicle control method, which causes the computer to recognize a sign portion corresponding to a lane reduction sign from a forward image captured by a camera that images the area around the vehicle, the sign portion including one or two folded line portions including a bent portion and a plurality of straight lines which are straight solid or dashed lines, and from the sign portion, obtain one or two first shapes corresponding to one or two of the folded line portions and a plurality of second shapes corresponding to the plurality of straight lines, and determine the position of the reducing lane based on the one or two first shapes and the plurality of second shapes. [Effects of the Invention]

[0010] According to the above embodiment, it is possible to provide a vehicle control device, a vehicle control method, and a control program that can accurately identify a reducing lane from a lane reduction sign. [Brief explanation of the drawing]

[0011] [Figure 1] Configuration diagram of a vehicle control device according to this embodiment [Figure 2] An explanatory diagram showing an example of a lane reduction sign. [Figure 3] Diagram of the labeling analysis unit [Figure 4] An explanatory diagram showing the region identified by the region discriminator from the marked area. [Figure 5] An explanatory diagram showing the bounding box identified by the rectangle classifier from the marked area. [Figure 6] Flowchart for identifying narrowing lanes [Figure 7] Flowchart of the first specific processing step [Figure 8] Flowchart of the second specific processing step [Figure 9] A photograph showing examples of various lane reduction signs. [Modes for carrying out the invention]

[0012] Hereinafter, embodiments of the vehicle control device, vehicle control method, and control program will be described with reference to the drawings.

[0013] Figure 1 shows the configuration of the vehicle control device 1. The vehicle control device 1 is provided in the vehicle 2. The vehicle 2 is a four-wheel automobile. The vehicle 2 may be an autonomous vehicle or a vehicle with a driving assistance function.

[0014] The vehicle 2 has a propulsion device 3, a braking device 4, and a steering device 5. The propulsion device 3 is a device that applies a driving force to the vehicle 2 and includes, for example, a power source and a transmission. The power source has at least one of an internal combustion engine such as a gasoline engine or a diesel engine and an electric motor. The braking device 4 is a device that applies a braking force to the vehicle 2 and includes, for example, a brake caliper that presses a pad against a brake rotor and an electric cylinder that supplies hydraulic pressure to the brake caliper. The steering device 5 is a device for changing the steering angle of the wheels and has, for example, a rack and pinion mechanism for steering the wheels and an electric motor for driving the rack and pinion mechanism. The propulsion device 3, the braking device 4, and the steering device 5 are controlled by the vehicle control device 1.

[0015] The vehicle 2 has an external recognition device 7. The external recognition device 7 is a device that detects external objects and the like. The external recognition device 7 is a sensor that captures electromagnetic waves and light from around the vehicle 2 to detect external objects and the like. The external recognition device 7 includes a radar 11, a lidar 12 (LIDAR), and a camera 13.

[0016] The radar 11 transmits radio waves around the vehicle 2 and detects the position and speed of an object by receiving the radio waves reflected by the object. The lidar 12 irradiates light such as infrared rays around the vehicle 2 and detects the position (distance and direction) of an object by capturing the reflected light. The lidar 12 may detect obstacles existing in the area in front of the vehicle 2.

[0017] Camera 13 images the surroundings of vehicle 2 and acquires an image of the surroundings of vehicle 2. The image of the surroundings of vehicle 2 includes surrounding vehicles, pedestrians, guardrails, curbs, walls, median strips, road shapes, lane lines, road signs, billboards, road markings drawn on the road, etc. that exist around vehicle 2. Camera 13 may be, for example, a digital camera using a solid-state imaging device such as a CCD or CMOS. Camera 13 includes a front camera that images at least the front area of vehicle 2. Camera 13 may also include a rear camera that images the rear of vehicle 2 and a pair of side cameras that image the left and right sides of vehicle 2. Camera 13 may be, for example, a stereo camera.

[0018] Vehicle 2 has vehicle sensor 15. Vehicle sensor 15 includes a vehicle speed sensor that detects the speed of vehicle 2, an acceleration sensor that detects acceleration, a yaw rate sensor that detects the angular velocity around the vertical axis, etc. Vehicle sensor 15 may also include an azimuth sensor that detects the orientation of vehicle 2.

[0019] Vehicle 2 has a GNSS (Global Navigation Satellite System) receiver 17. GNSS receiver 17 identifies the position (latitude and longitude) of vehicle 2 based on signals received from artificial satellites (positioning satellites).

[0020] Vehicle 2 has an HMI 18 (Human Machine Interface). HMI 18 notifies various information to the occupant by display and voice and accepts input operations by the occupant. HMI 18 includes a display 21 and a speaker 22. Display 21 may be a touch panel display.

[0021] The vehicle control device 1 is a computer having a processor 31 and a memory 32 that is communicatively connected to the processor 31. The processor 31 may include at least one of the following as its core: a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a RISC (Reduced Instruction Set Computer). The memory 32 stores control programs executed by the processor 31 and various data. The memory 32 may include at least one of volatile memory and non-volatile memory. The volatile memory may be, for example, DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). The non-volatile memory may be an SSD (Solid State Drive), flash memory, magnetic disk storage device, or optical disk storage device. At least a part of the vehicle control device 1 may be implemented by hardware such as an LSI (Large Scale Integration), ASIC (application specific integrated circuit), or FPGA (field-programmable gate array), or by a combination of software and hardware. The vehicle control device 1 may be composed of a single piece of hardware, or it may be composed of multiple pieces of hardware that can communicate with each other. Part of the vehicle control device 1 may be composed of an external server located outside the vehicle 2.

[0022] The processor 31 implements various applications by executing programs stored in memory 32. Programs may be stored on removable recordable media such as DVDs or CD-ROMs, and installed in memory 32 when the recordable media is read by a reader. Alternatively, programs may be downloaded to and installed in memory 32 via a communication network such as the internet.

[0023] The memory 32 should preferably store map information. The map information should preferably be high-resolution map information. The map information should include road information such as the type of road (expressway, toll road, national road, prefectural road), the number of lanes on the road, the center position of each lane (3D coordinates including longitude, latitude, and height), the shape of road markings such as road markings and lane boundaries, the presence or absence of sidewalks, curbs, fences, etc., the location of intersections, the locations of lane merging and branching points, the area of ​​emergency parking zones, the width of each lane, and road signs. In addition, the map information may also include traffic regulation information, address information (address and postal code), facility information, telephone number information, etc.

[0024] The processor 31 functions as an obstacle recognition unit 41, a vehicle position recognition unit 42, a traffic sign analysis unit 43, a driving lane determination unit 44, a notification unit 45, and a driving support unit 46 by executing programs stored in the memory 32.

[0025] The obstacle recognition unit 41 recognizes the surrounding environment of the vehicle 2. Based on the detection results of the external environment recognition device 7, the obstacle recognition unit 41 recognizes the surrounding environment (external world), including obstacles located around the vehicle 2, the shape of the road, the presence or absence of sidewalks, road markings, etc. Obstacles include, for example, guardrails, utility poles, surrounding vehicles, and people such as pedestrians. From the detection results of the external environment recognition device 7, the obstacle recognition unit 41 can acquire the status of surrounding vehicles, such as their position, speed, and acceleration.

[0026] The vehicle position recognition unit 42 recognizes the position of vehicle 2. The vehicle position recognition unit 42 may recognize the position of vehicle 2 based on the GNSS signal received by the GNSS receiver 17.

[0027] The sign analysis unit 43 recognizes the lane reduction sign 50 and identifies the location of the reducing lane from the lane reduction sign 50. Figure 3 shows a typical example of the lane reduction sign 50. The lane reduction sign 50 may be drawn on a sign or billboard by itself, or it may be drawn on a sign or billboard together with other characters or figures.

[0028] As shown in Figure 2, the lane reduction sign 50 includes one or two folded sections 51, including a bent section 51A, and a plurality of straight sections 52, which are straight solid or dashed lines. The folded section 51 has an upper section 51B and a lower section 51C that extend linearly vertically, and a straight intermediate section 51D that connects the upper end of the lower section 51C and the lower end of the upper section 51B and extends inclined with respect to the vertical direction. The intermediate section 51D constitutes the bent section 51A. The upper section 51B is offset to the left or right relative to the lower section 51C. The straight sections 52 include at least one dashed section 53 and zero or one solid section 54. Each of the dashed section 53 and the solid section 54 extends vertically. Each of the lower section 51C, dashed section 53, and solid section 54 of the folded section 51 extends parallel to each other with a distance between them in the left-right direction. Note that if the sign indicates a decrease or increase in road width rather than a decrease in the number of lanes, the dashed line section 53 may not be included.

[0029] A folded line section 51 or a solid line section 54 is placed at the left and right ends of the lane reduction sign 50. If a solid line section 54 is placed at one of the left or right ends of the lane reduction sign 50, a folded line section 51 is placed at the other end. A folded line section 51 may be placed at both the left and right ends of the lane reduction sign 50 (see Figure 9).

[0030] At least one dashed line section 53 is positioned between the folded line sections 51 or solid line sections 54 located at the left and right ends of the lane reduction sign 50. The dashed line section 53 includes a long dashed line 53A and a short dashed line 53B whose vertical length is less than or equal to a predetermined value relative to the vertical length of the long dashed line 53A. The predetermined value is, for example, 70%. The lane reduction sign 50 includes at least one short dashed line 53B. The lane reduction sign 50 may or may not include a long dashed line 53A.

[0031] The areas between the folded line section 51, the dashed line section 53, and the solid line section 54 represent lanes. The left and right widths of the areas between the folded line section 51, the dashed line section 53, and the solid line section 54 are set to be approximately equal. The number of lanes is the sum of the dashed line sections 53 plus 1. Also, the number of lanes is the sum of the folded line sections 51, the dashed line sections 53, and the solid line sections 54 minus 1. Lanes include narrowing lanes and remaining lanes (continuing lanes). Narrowing lanes have an end within a predetermined range ahead. On the other hand, remaining lanes do not have an end within a predetermined range ahead. The folded line section 51 or the short dashed line 53B indicates the location where narrowing lanes exist. The number of short dashed lines 53B indicates the number of narrowing lanes. The area located below the middle section 51D of the folded line section 51 represents narrowing lanes. The number of short dashed lines 53B located below the middle section 51D and upper section 51B of the folded line section 51, which is located at the left or right end of the lane reduction sign 50, represents the number of reducing lanes on the left or right side. For example, if there are two areas separated by short dashed lines 53B below the middle section 51D of the folded line section 51, then there are two reducing lanes. The vertical length of the area representing the reducing lanes is shorter than the vertical length of the area representing the remaining lanes.

[0032] As shown in Figure 3, the sign analysis unit 43 includes a sign recognition unit 60 and a lane reduction identification unit 61. The sign recognition unit 60 recognizes a sign portion 80 corresponding to a lane reduction sign 50 from an image captured by a camera 13 that images the area around the vehicle 2. As shown in Figures 4 and 5, the sign recognition unit 60 may recognize the sign portion 80 corresponding to the lane reduction sign 50 from an image using various known image recognition techniques. For example, the sign recognition unit 60 may recognize the sign portion 80 corresponding to the lane reduction sign 50 using a trained model that outputs the sign portion 80 corresponding to the lane reduction sign 50 for an input image. The trained model may be composed of, for example, a convolutional neural network (CNN). Alternatively, the sign recognition unit 60 may recognize the sign portion 80 corresponding to the lane reduction sign 50 by performing image pattern matching using dictionary information that stores various lane reduction signs 50.

[0033] As shown in Figures 4 and 5, the reducing lane identification unit 61 obtains one or two first shapes 81A, 83A corresponding to one or two folded line sections 51 and a plurality of second shapes 81B, 83B corresponding to a plurality of straight sections 52 from the sign section 80, and identifies the location of the reducing lane based on the one or two first shapes 81A, 83A and the plurality of second shapes 81B, 83B. As shown in Figure 3, the reducing lane identification unit 61 has at least one of a first identification unit 63 and a second identification unit 64, and a determination unit 65 that identifies the reducing lane based on at least one of the identification results of the first identification unit 63 and the identification results of the second identification unit 64. In this embodiment, the reducing lane identification unit 61 has a first identification unit 63 and a second identification unit 64. The reducing lane identification unit 61 also has a third identification unit 66.

[0034] The first identification unit 63 includes a region discriminator 68 and a first post-processing unit 69. The first identification unit 63 identifies the location of the narrowing lane by executing a first identification process (method). As shown in Figure 4, the region discriminator 68 acquires multiple regions 81 from the marked portion 80 that correspond to one or two polyline portions 51 and multiple straight portions 52. The region discriminator 68 performs clustering on the marked portion 80 acquired by the marked analysis unit 43 and recognizes each region 81 from the marked portion 80 that corresponds to the polyline portions 51 and straight portions 52, respectively. The clustering method used here may be, for example, DBSCAN (Density-Based Spatial Clustering of Applications with Noise). The region discriminator 68 may be adjusted to recognize a linearly extending dashed line portion 53 as one region 81. The region discriminator 68 identifies the coordinates of each region 81 in order to identify the location and shape of each region 81.

[0035] The first post-processing unit 69 identifies the multiple regions 81 acquired by the region discriminator 68 as one or two first shapes 81A corresponding to one or two polyline sections 51, and multiple second shapes 81B corresponding to multiple straight lines 52. The first shapes 81A and second shapes 81B may be identified based on the left-right width of the region 81, i.e., its length in the left-right direction. The left-right width of each region 81 may be acquired based on the coordinates of the left and right ends of each region 81. The first shapes 81A corresponding to the polyline sections 51 have a larger left-right width than the second shapes 81B corresponding to the straight lines 52. For example, the first post-processing unit 69 divides each region 81 into a first group with a relatively large left-right width and a second group with a relatively small left-right width, and identifies each region 81 in the first group as a first shape 81A and each region 81 in the second group as a second shape 81B.

[0036] The first after-processing unit 69 determines that the leftmost lane is a reducing lane if there is one first shape 81A and the first shape 81A is located at the leftmost end. The first after-processing unit 69 also determines that the rightmost lane is a reducing lane if there is one first shape 81A and the first shape 81A is located at the rightmost end. The first after-processing unit 69 determines that both the leftmost lane and the rightmost lane are reducing lanes if there are two first shapes 81A.

[0037] The first post-processing unit 69 identifies a second shape 81B among a plurality of second shapes 81B whose vertical length is less than or equal to a predetermined value relative to the vertical length of the first shape 81A as a short second shape 81C. The predetermined value may be, for example, 70%. The first post-processing unit 69 determines the number of reducing lanes based on the number of short second shapes 81C present in the left-right width of the first shape 81A. The number of reducing lanes is the number of short second shapes 81C present in the left-right width of the first shape 81A. For example, if the first shape 81A is located at the left end and two short second shapes 81C are located within the left-right width of the first shape 81A, the first post-processing unit 69 determines that the first and second lanes from the left are reducing lanes. For example, if the first shape 81A is located at the right end and one short second shape 81C is located within the left-right width of the first shape 81A, the first post-processing unit 69 determines that the first lane from the right is a reducing lane.

[0038] The first after-processing unit 69 performs an operation to subtract 1 from the total number of first shapes 81A and second shapes 81B, and obtains the calculated value as the number of lanes before the lane reduction. The first after-processing unit 69 may also identify the remaining lanes by removing the reducing lane from the number of lanes before the lane reduction.

[0039] As shown in Figure 3, the first post-processing unit 69 outputs a determination result, including the location of the narrowing lane, to the determination unit 65. The data output from the first post-processing unit 69 to the determination unit 65 may include, for example, lane numbers assigned in ascending order from left to right for each lane before the lane narrowing, and a flag indicating that the lane corresponding to the lane number is the narrowing lane.

[0040] The second identification unit 64 includes a rectangle discriminator 71 and a second post-processing unit 72. The second identification unit 64 identifies the position of the narrowing lane by executing a second identification process (method). As shown in Figure 5, the rectangle discriminator 71 obtains multiple bounding boxes 83 from the marked portion 80, each enclosing one or two polyline portions 51 and multiple straight line portions 52. The rectangle discriminator 71 may be composed of a trained model using R-CNN (Region-Convolutional Neural Networks), Fast R-CNN, Faster R-CNN, YOLO, SSD (Single Shot Detector), etc. Each bounding box 83 is a rectangle extending in the left-right and up-down directions. Each bounding box 83 may be represented, for example, by the coordinates of two corner points located diagonally opposite each other, or by the coordinates of the center point, width, and height. The rectangle discriminator 71 is preferably adjusted so that the linearly extending dashed line portion 53 sets one bounding box 83.

[0041] The second post-processing unit 72 identifies the multiple bounding boxes 83 acquired by the rectangle discriminator 71 as one or two first shapes 83A corresponding to one or two folded line portions 51, and multiple second shapes 83B corresponding to multiple straight line portions 52. The first shapes 83A and second shapes 83B may be identified based on the left-right width of the bounding box 83. The first shapes 83A corresponding to the folded line portions 51 have a larger left-right width than the second shapes 83B corresponding to the straight line portions 52. For example, the second post-processing unit 72 divides each bounding box 83 into a first group with a relatively large left-right width and a second group with a relatively small left-right width, and identifies each bounding box 83 in the first group as a first shape 83A and each bounding box 83 in the second group as a second shape 83B.

[0042] The second after-processing unit 72 determines that the leftmost lane is a reducing lane when there is one first shape 83A and the first shape 83A is located at the leftmost end. The second after-processing unit 72 also determines that the rightmost lane is a reducing lane when there is one first shape 83A and the first shape 83A is located at the rightmost end. The second after-processing unit 72 determines that both the leftmost lane and the rightmost lane are reducing lanes when there are two first shapes 83A.

[0043] The second post-processing unit 72 identifies among a plurality of second shapes 83B as short second shapes 83C if their vertical length is less than or equal to a predetermined value relative to the vertical length of the first shape 83A. The predetermined value may be, for example, 70%. The second post-processing unit 72 determines the number of reducing lanes based on the number of short second shapes 83C present in the left-right width of the first shape 83A. The number of reducing lanes is the number of short second shapes 83C present in the left-right width of the first shape 83A. For example, if the first shape 83A is located at the left end and two short second shapes 83C are located within the left-right width of the first shape 83A, the second post-processing unit 72 determines that the first and second lanes from the left are reducing lanes. For example, if the first shape 83A is located at the right end and one short second shape 83C is located within the left-right width of the first shape 83A, the second post-processing unit 72 determines that the first lane from the right is a reducing lane.

[0044] The second post-processing unit 72 subtracts 1 from the total number of first shapes 83A and second shapes 83B, and obtains the calculated value as the number of lanes before the lane reduction. The second post-processing unit 72 may also identify the remaining lanes by removing the reducing lane from the number of lanes before the lane reduction.

[0045] The second post-processing unit 72 outputs a determination result, including the location of the narrowing lane, to the determination unit 65. The data output from the second post-processing unit 72 to the determination unit 65 may include, for example, lane numbers assigned in ascending order from left to right for each lane before the lane narrowing, and a flag indicating that the lane corresponding to the lane number is the narrowing lane.

[0046] As shown in Figure 3, the third identification unit 66 has a classifier 74 and uses the classifier 74 to identify the location of the narrowing lane in response to the input of the sign portion 80. The third identification unit 66 performs a third identification process using the classifier 74. The classifier 74 outputs the location of the narrowing lane in response to the input of the sign portion 80. The classifier 74 may be a trained model that outputs the location of the narrowing lane in response to the input of the sign portion 80. The trained model may be composed of a folded neural network. The trained model may be trained based on training data that associates an image of the lane reduction sign 50 with the number of lanes before the lane reduction and the location of the narrowing lane. The third identification unit 66 outputs an identification result, including the location of the narrowing lane, to the decision unit 65 in response to the input of the sign portion 80 of the image. The data output from the third post-processing unit to the decision unit 65 may include, for example, lane numbers assigned in ascending order from left to right for each lane before the lane reduction, and a flag indicating that the lane corresponding to the lane number is a narrowing lane.

[0047] The decision unit 65 determines the location of the narrowing lane based on the determination results of the first identification unit 63, the second identification unit 64, and the third identification unit 66. The decision unit 65 may perform a majority vote on the determination results of the first identification unit 63, the second identification unit 64, and the third identification unit 66, and determine the location of the narrowing lane based on the most frequently occurring determination result. Alternatively, the majority vote may be performed for each lane of each determination result. The decision unit 65 may also pre-set priority levels for the determination results of the first identification unit 63, the second identification unit 64, and the third identification unit 66, and determine the location of the narrowing lane based on the determination result with the highest priority. The decision unit 65 outputs narrowing lane information to the driving lane determination unit 44. The narrowing lane information output by the decision unit 65 can also be considered the output of the narrowing lane identification unit 61 and the lane analysis unit. The lane reduction information may include, for example, lane numbers assigned in ascending order from left to right for each lane before the lane reduction, and a flag indicating that the lane corresponding to the lane number is a reducing lane.

[0048] Next, with reference to Figures 6 to 8, the control flow of the lane reduction identification process performed by the sign analysis unit 43 will be described. The sign analysis unit 43 repeatedly performs the lane reduction identification process at predetermined time intervals. First, the sign recognition unit 60 of the sign analysis unit 43 recognizes the sign portion 80 corresponding to the lane reduction sign 50 from the image of the front of the vehicle 2 acquired by the camera 13 (ST1).

[0049] Next, the sign analysis unit 43 determines whether or not a sign portion 80 corresponding to the lane reduction sign 50 exists in the image (ST2). If a sign portion 80 corresponding to the lane reduction sign 50 does not exist in the image (ST2: No), the sign analysis unit 43 terminates the lane reduction identification process.

[0050] If a sign portion 80 corresponding to a lane reduction sign 50 exists in the image (ST2: Yes), the first identification unit 63 performs a first identification process using the region discriminator 68 (ST3).

[0051] The first identification process is performed based on the flowchart in Figure 7. In the first identification process, first, the region discriminator 68 acquires multiple regions 81 corresponding to the polyline portion 51 and the straight line portion 52 from the marked portion 80 of the image (ST11). Next, the first post-processing unit 69 separates the multiple regions 81 into a first shape 81A or a second shape 81B (ST12). Next, the first post-processing unit 69 identifies a short second shape 81C from among the multiple second shapes 81B (ST13).

[0052] Next, the first post-processing unit 69 determines whether or not an error exists in the identification of the first shape 81A, the second shape 81B, and the short second shape 81C (ST14). The first post-processing unit 69 may determine that an error exists if any of the following conditions are met: (1) the number of first shapes 81A is 0 or 3 or more, (2) the total number of first shapes 81A and second shapes 81B is 2 or less, or (3) the number of short second shapes 81C is 0.

[0053] If there are no errors in identifying the first shape 81A, the second shape 81B, and the short second shape 81C (ST14: No), the first post-processing unit 69 sets the number of lanes before the lane reduction to the number of regions 81 minus 1 (ST15).

[0054] Next, the first after-processing unit 69 determines the number of left-side sidings (ST16). At this time, the first after-processing unit 69 determines whether or not a first shape 81A exists at the left end of the multiple regions 81. If no first shape 81A exists at the left end of the multiple regions 81, the first after-processing unit 69 determines that the number of left-side sidings is 0. If a first shape 81A exists at the left end of the multiple regions 81, the first after-processing unit 69 measures the number of short second shapes 81Cs that exist within the width of the first shape 81A at the left end and sets that number as the number of left-side sidings.

[0055] Next, the first post-processing unit 69 identifies the number of right-side reducing lanes (ST17). At this time, the first post-processing unit 69 determines whether or not a first shape 81A exists at the right end of the multiple regions 81. If no first shape 81A exists at the right end of the multiple regions 81, the first post-processing unit 69 identifies the number of right-side reducing lanes as 0. If a first shape 81A exists at the right end of the multiple regions 81, the first post-processing unit 69 measures the number of short second shapes 81Cs that exist within the left-right width of the first shape 81A at the right end and sets that number as the number of right-side reducing lanes. After step ST16, the first identification process ends.

[0056] Next, the first post-processing unit 69 generates a specific result (ST18) based on the results of steps ST16 and ST17. The specific result may include lane numbers assigned in ascending order from left to right for each lane before the lane reduction, and a flag indicating that the lane corresponding to the lane number is a reducing lane.

[0057] If an error occurs in identifying the first shape 81A, the second shape 81B, or the short second shape 81C (ST14: Yes), the first post-processing unit 69 creates an identification result indicating that the reducing lane cannot be identified (ST19). After the identification result is created in step ST18 or ST19, the first identification process ends.

[0058] After the first identification process is performed, the second identification unit 64 performs a second identification process using the rectangle discriminator 71 (ST4).

[0059] The second identification process is performed based on the flowchart in Figure 8. In the second identification process, first, the rectangle discriminator 71 obtains a plurality of bounding boxes 83 corresponding to the polyline portion 51 and the straight line portion 52 from the marked portion 80 of the image (ST21). Next, the second post-processing unit 72 sorts the plurality of bounding boxes 83 into a first shape 83A or a second shape 83B (ST22). Then, the second post-processing unit 72 identifies a short second shape 83C from among the plurality of second shapes 83B (ST23).

[0060] Next, the second post-processing unit 72 determines whether or not there is an error in identifying the first shape 83A, the second shape 83B, and the short second shape 83C (ST24). The second post-processing unit 72 may determine that an error exists if any of the following conditions are met, for example: (1) the number of first shapes 83A is 0 or 3 or more, (2) the total number of first shapes 83A and second shapes 83B is 2 or less, or (3) the number of short second shapes 83C is 0.

[0061] If there are no errors in identifying the first shape 83A, the second shape 83B, and the short second shape 83C (ST24: No), the second post-processing unit 72 sets the number of bounding boxes 83 minus 1 to the number of lanes before the lane reduction (ST25).

[0062] Next, the second post-processing unit 72 determines the number of left-side sidings (ST26). At this time, the second post-processing unit 72 determines whether or not a first shape 83A exists at the left end of the multiple bounding boxes 83. If no first shape 83A exists at the left end of the multiple bounding boxes 83, the second post-processing unit 72 determines that the number of left-side sidings is 0. If a first shape 83A exists at the left end of the multiple bounding boxes 83, the second post-processing unit 72 measures the number of short second shapes 83Cs that exist within the width of the first shape 83A at the left end and sets that number as the number of left-side sidings.

[0063] Next, the second post-processing unit 72 identifies the number of right-side sidings (ST27). At this time, the second post-processing unit 72 determines whether or not a first shape 83A exists at the right end of the multiple bounding boxes 83. If no first shape 83A exists at the right end of the multiple bounding boxes 83, the second post-processing unit 72 identifies the number of right-side sidings as 0. If a first shape 83A exists at the right end of the multiple bounding boxes 83, the second post-processing unit 72 measures the number of short second shapes 83Cs that exist within the width of the first shape 83A at the right end and sets that number as the number of right-side sidings. After step ST26, the second identification process ends.

[0064] Next, the second post-processing unit 72 creates a specific result (ST28) based on the results of steps ST26 and ST27. The specific result may include lane numbers assigned in ascending order from left to right for each lane before the lane reduction, and a flag indicating that the lane corresponding to the lane number is a reducing lane.

[0065] If an error occurs in identifying the first shape 83A, the second shape 83B, or the short second shape 83C (ST24: Yes), the second post-processing unit 72 creates an identification result indicating that the reducing lane cannot be identified (S29). After the identification result is created in step ST28 or ST29, the second identification process ends.

[0066] After the second identification process is performed, the third identification unit 66 performs a third identification process using the classifier 74 (ST5). In the third identification process, the classifier 74 outputs an identification result, including the position of the narrowing lane, in response to the input of the marked portion 80 of the image.

[0067] After the third identification process is performed, the determination unit 65 determines the location of the narrowing lane based on the identification results of the first identification unit 63, the identification results of the second identification unit 64, and the identification results of the third identification unit 66 (ST6).

[0068] The driving lane determination unit 44 determines whether the lane in which vehicle 2 is traveling is a narrowing lane, based on the narrowing lane information from the narrowing lane identification unit 61 and the position of vehicle 2. The driving lane determination unit 44 also identifies the lane in which vehicle 2 is traveling, based on the position of vehicle 2 acquired by the vehicle position recognition unit 42 and map information. The driving lane determination unit 44 determines whether the lane in which vehicle 2 is traveling is a narrowing lane by comparing the lane in which vehicle 2 is traveling with the position of the narrowing lane included in the narrowing lane information.

[0069] The notification unit 45 controls at least one of the notification devices, the display 21 and the speaker 22, in order to notify the occupants when the lane determination unit 44 determines that the lane in which the vehicle 2 is traveling is a narrowing lane. The notification unit 45 may control the display 21 to display text or images on the display 21 to encourage a lane change. The notification unit 45 may also control the speaker 22 to output voice or sound effects to encourage a lane change.

[0070] The driver assistance unit 46 controls the steering device 5 to change the vehicle 2 to a different remaining lane when the lane determination unit 44 determines that the lane in which the vehicle 2 is traveling is a narrowing lane. The driver assistance unit 46 identifies the remaining lane based on the narrowing lane information. The driver assistance unit 46 then sets a target trajectory for changing the vehicle 2 to the remaining lane. At this time, the driver assistance unit 46 may set the target trajectory such that the time to collision (TTC) with an obstacle recognized by the obstacle recognition unit 41 is greater than or equal to a predetermined value. The driver assistance unit 46 may then control the steering device 5 so that the position of the vehicle 2 follows the target trajectory. In other embodiments, the driver assistance unit 46 may also control the steering wheel for operating the steering device 5 to provide assist force so that the position of the vehicle 2 follows the target trajectory.

[0071] According to the above embodiment, a vehicle control device 1 can be provided that can accurately identify the position of the reducing lane from the lane reduction sign 50. Since the vehicle control device 1 identifies the position of the reducing lane based on the characteristic parts of the lane reduction sign 50, it can appropriately identify the position of the reducing lane for various variations of the lane reduction sign 50.

[0072] The first identification process performed by the first identification unit 63 and the second identification process performed by the second identification unit 64 identify the position of the reducing lane based on the characteristic parts of the lane reduction sign 50, so that the position of the reducing lane can be accurately identified even if the lane reduction sign 50 has not been learned. Specifically, the first identification unit 63 and the second identification unit 64 identify the position of the reducing lane based on the positional relationship between the folded line section 51 and the straight section 52, which are elements of the lane reduction sign 50, so that they can handle various lane reduction signs 50 in which the number of folded line sections 51 and straight sections 52 has changed. In addition, the vehicle control device 1 can improve the accuracy of identifying the position of the reducing lane by also considering the identification result of the third identification unit 66, which includes the classifier 74.

[0073] When the vehicle 2 is located in a narrowing lane, the vehicle control device 1 can control at least one of the display 21 and the speaker 22 to alert the occupants. Furthermore, when the vehicle 2 is located in a narrowing lane, the vehicle control device 1 can control the steering device 5 to automatically change the lane of the vehicle 2.

[0074] The embodiments are not limited to the above configuration and can be broadly modified. For example, the first specification section 63, second specification section 64, third specification section 66, and part of the determination section 65 of the reducing lane specification section 61 may be omitted. For example, the reducing lane specification section 61 may be composed of the first specification section 63 or the second specification section 64. Alternatively, the reducing lane specification section 61 may be composed of the first specification section 63, the second specification section 64, and the determination section 65. Alternatively, the reducing lane specification section 61 may be composed of the first specification section 63, the third specification section 66, and the determination section 65. Alternatively, the reducing lane specification section 61 may be composed of the second specification section 64, the third specification section 66, and the determination section 65.

[0075] The first after-processing unit 69 does not need to distinguish between the second shape 81B and the short second shape 81C. In this case, the first after-processing unit 69 may determine the number of reducing lanes based on the number of second shapes 81B present in the left-right width of the first shape 81A. Similarly, the second after-processing unit 72 does not need to distinguish between the second shape 83B and the short second shape 83C. In this case, the second after-processing unit 72 may determine the number of reducing lanes based on the number of second shapes 83B present in the left-right width of the first shape 83A.

[0076] The lane reduction identification unit 61 may identify lane reductions at predetermined time intervals and correct the current identification result based on the previous identification result. This improves the accuracy of identifying the location of lane reductions. Since images of lane reduction signs 50 are acquired while the vehicle 2 is in motion, the acquired images of lane reduction signs 50 may sometimes be partially missing or unclear. Therefore, the accuracy of identifying the location of lane reductions can be improved by having the lane reduction identification unit 61 identify lane reductions multiple times at predetermined time intervals.

[0077] The lane reduction identification section 61 may also identify a decrease or increase in width based on the first shapes 81A and 83A and the second shapes 81B and 83B. For example, the lane reduction identification section 61 may identify a sign as indicating a decrease or increase in width if there is no short dashed line 53B in the sign. The lane reduction identification section 61 may also identify a decrease or increase in width based on the shapes of the first shapes 81A and 83A. For example, the lane reduction identification section 61 may identify a decrease or increase in width based on the left-right positions of the upper and lower ends of the first shapes 81A and 83A.

[0078] The above embodiments may also be described as follows:

[0079] One embodiment includes a vehicle control device 1 which recognizes a sign portion 80 corresponding to a lane reduction sign 50 from an image captured by a camera 13 that images the area around a vehicle 2, the sign portion 80 which includes one or two folded line portions 51 including a bent portion 51A and a plurality of straight line portions 52 which are straight solid or dashed lines, and a lane reduction identification unit 61 which obtains one or two first shapes 81A, 83A corresponding to one or two of the folded line portions 51 and a plurality of second shapes 81B, 83B corresponding to the plurality of straight line portions 52 from the sign portion 80, and identifies the position of the reducing lane based on one or two of the first shapes and the plurality of second shapes.

[0080] According to this embodiment, a vehicle control device 1 can be provided that can accurately identify the location of the reducing lane from the lane reduction sign 50. Since the vehicle control device 1 identifies the location of the reducing lane based on the characteristic parts of the lane reduction sign 50, it can appropriately identify the location of the reducing lane for various variations of the lane reduction sign 50.

[0081] In the above embodiment, the reducing lane identification unit 61 has at least one of a first identification unit 63 and a second identification unit 64, and a determination unit 65 that identifies the reducing lane based on at least one of the identification results of the first identification unit 63 and the identification results of the second identification unit 64, and the first identification unit 63 has an area discriminator 68 that acquires a plurality of areas 81 corresponding to one or two of the folded line sections 51 and a plurality of the straight line sections 52 from the indicator portion 80, and discriminates the plurality of areas 81 acquired by the area discriminator 68 as one or two of the first shapes corresponding to one or two of the folded line sections 51 and a plurality of the second shapes corresponding to a plurality of the straight line sections 52, and one or two The first post-processing unit 69 identifies the position of the reducing lane based on the first shape and the plurality of second shapes, and the second identifying unit 64 may include a rectangle discriminator 71 that acquires a plurality of bounding boxes 83 surrounding one or two of the folded line portions 51 and a plurality of the straight line portions 52 from the marking portion 80, and a second post-processing unit 72 that discriminates the plurality of bounding boxes 83 acquired by the rectangle discriminator 71 as the first shape corresponding to one or two of the folded line portions 51 and a plurality of second shapes corresponding to a plurality of the straight line portions 52, and identifies the position of the reducing lane based on the one or two first shapes and the plurality of second shapes.

[0082] According to this embodiment, the vehicle control device 1 identifies the position of the reducing lane based on the characteristic parts of the lane reduction sign 50, and therefore can appropriately identify the position of the reducing lane for various variations of the lane reduction sign 50.

[0083] In the above embodiment, the reducing lane identification unit 61 has a classifier 74 that outputs the position of the reducing lane in response to the input of the sign portion 80, and a third identification unit 66 that uses the classifier 74 to identify the position of the reducing lane in response to the input of the sign portion 80, and the determination unit 65 may identify the position of the reducing lane based on at least one of the identification result of the first identification unit 63 and the identification result of the second identification unit 64 and the identification result of the third identification unit 66.

[0084] According to this embodiment, the identification result of the third identification unit 66 using the classifier 74 is taken into consideration, thereby improving the accuracy of identifying the location of the narrowing lane.

[0085] In the above embodiment, the reducing lane identification unit 61 may identify the position of the reducing lane based on the positions of one or two of the first shapes relative to a plurality of the second shapes, and the positions of the second shapes among the plurality of the second shapes whose vertical length is less than or equal to a predetermined value relative to the vertical length of the first shape.

[0086] According to this embodiment, the vehicle control device 1 identifies the position of the reducing lane based on the characteristic parts of the lane reduction sign 50, and therefore can appropriately identify the position of the reducing lane for various variations of the lane reduction sign 50.

[0087] In the above embodiment, there may be a lane determination unit 44 that determines whether the lane in which the vehicle 2 is traveling is the aforementioned lane, based on the lane determination information relating to the lane determination output by the lane determination unit 61 and the position of the vehicle 2.

[0088] According to this embodiment, it is possible to determine whether or not vehicle 2 needs to change lanes.

[0089] In the above embodiment, the lane determination unit 44 may also have a notification unit 45 that controls a notification device (display 21, speaker 22) to notify the occupants when it determines that the lane in which the vehicle 2 is traveling is the narrowing lane.

[0090] In this embodiment, the vehicle control device 1 can control the notification device to inform the driver of the vehicle 2 that a lane change is necessary.

[0091] In the above embodiment, if the lane determination unit 44 determines that the lane in which the vehicle 2 is traveling is the narrowing lane, the driving support unit 46 may have a steering support unit 46 that controls the steering device 5 to cause the vehicle 2 to change lanes to a different remaining lane.

[0092] In this embodiment, the vehicle control device 1 can control the steering device 5 to change the lane of the vehicle 2.

[0093] In the above embodiment, the lane reduction unit 61 may identify the lane reduction at predetermined time intervals and correct the current identification result based on the previous identification result.

[0094] According to this embodiment, the accuracy of identifying the location of the narrowing lane can be improved.

[0095] Another embodiment is a vehicle 2 control method performed by a computer, in which a sign portion 80 corresponding to a lane reduction sign 50 is recognized from a forward image captured by a camera 13 that images the area around the vehicle 2, the sign portion 80 being corresponding to a lane reduction sign 50 being corresponding to a lane reduction sign 50 being corresponding to a lane reduction sign 50 being corresponding to a lane reduction sign 50 being corresponding to a lane reduction sign 50 being corresponding to a lane reduction sign 50 being recognized from a forward image captured by a camera 13 that images the area around the vehicle 2 being recognized from the sign portion 80 being corresponding to a lane reduction sign 50 being recognized to obtain from the sign portion 80 one or two first shapes 81A, 83A corresponding to the lane reduction sign 51 and a lane reduction sign being corresponding to a lane reduction sign being recognized from a forward image captured by a camera 13 that images the area around the vehicle 2 being recognized from a lane reduction sign 50 being corresponding to a lane reduction sign 50 being corresponding to a lane reduction sign 50 being corresponding to a lane reduction sign 50 being recognized from a lane reduction sign 50 being corresponding to a lane reduction sign 50 being corresponding to a lane reduction sign 50 being corresponding to a lane reduction sign 50 being recognized from a forward image captured by a camera 13 that images the area around the vehicle 2 being recognized from a lane reduction sign 50 being corresponding to

[0096] According to this embodiment, a vehicle control method can be provided that can accurately identify the position of the reducing lane from the lane reduction sign 50. Since the vehicle control method identifies the position of the reducing lane based on the characteristic parts of the lane reduction sign 50, it can appropriately identify the position of the reducing lane for various variations of the lane reduction sign 50.

[0097] Another embodiment is a control program for causing a computer to execute a vehicle 2 control method, which causes the computer to recognize a sign portion 80 corresponding to a lane reduction sign 50, which includes one or two folded line portions 51 including a bent portion 51A and a plurality of straight line portions 52 that are straight solid or dashed lines, from a forward image captured by a camera 13 that captures the area around the vehicle 2, and to obtain one or two first shapes 81A, 83A corresponding to one or two of the folded line portions 51 and a plurality of second shapes 81B, 83B corresponding to the plurality of straight line portions 52 from the sign portion 80, and to determine the position of the reducing lane based on one or two of the first shapes 81A, 83A and the plurality of second shapes 81B, 83B.

[0098] According to this embodiment, it is possible to provide a control program for executing a vehicle control method that can accurately identify the position of a reducing lane from a lane reduction sign 50. [Explanation of symbols]

[0099] 1: Vehicle control system 2: Vehicles 5: Steering gear 13: Camera 21: Display 22: Speaker 41: Obstacle Recognition Unit 42: Vehicle position recognition unit 43: Label analysis department 44: Lane determination unit 45: Hochi Department 46: Driver Support Department 50: Lane reduction sign 60: Sign recognition section 61: Designated area for reducing lanes 63:1st specific part 64:Second specific part 65: Decision Section 66:Third Specific Part 68: Area classifier 69: First Post-Processing Unit 71: Rectangle classifier 72: Second post-processing unit 74 :Classifier 80: Sign part 81 :Area 81A: First shape 81B: 2nd shape 81C: Short 2nd shape 83: Bounding Box 83A: 1st shape 83B: 2nd shape 83C: Short 2nd shape

Claims

1. A vehicle control device, A sign recognition unit recognizes a sign portion corresponding to a lane reduction sign, which includes one or two folded line sections including a bend, and multiple straight sections that are solid or dashed lines, from an image captured by a camera that captures the area around the vehicle. A vehicle control device having a lane-reducing unit that obtains one or two first shapes corresponding to one or two of the folded line portions and a plurality of second shapes corresponding to a plurality of the straight sections from the marking portion, and determines the position of the lane-reducing lane based on one or two of the first shapes and the plurality of second shapes.

2. The aforementioned lane reduction designation section is, At least one of the first and second specified sections, The system includes a determination unit that identifies the reducing lane based on the identification result of the first identification unit and at least one of the identification results of the second identification unit, The first identification unit includes a region discriminator that acquires a plurality of regions from the marking portion corresponding to one or two of the polyline portions and a plurality of the straight lines, and a first post-processing unit that discriminates the plurality of regions acquired by the region discriminator as one or two of the first shapes corresponding to one or two of the polyline portions and a plurality of the second shapes, and identifies the position of the reducing lane based on the one or two of the first shapes and the plurality of the second shapes. The vehicle control device according to claim 1, wherein the second identifying unit includes a rectangle discriminator that acquires a plurality of bounding boxes from the marking portion, each surrounding one or two of the polyline portions and a plurality of the straight sections, and a second post-processing unit that discriminates the plurality of bounding boxes acquired by the rectangle discriminator as a first shape corresponding to one or two of the polyline portions and a plurality of second shapes corresponding to a plurality of the straight sections, and identifies the position of the reducing lane based on the one or two first shapes and the plurality of second shapes.

3. The aforementioned lane reduction designation section is, The system includes a classifier that outputs the position of the reducing lane in response to the input of the sign portion, and a third identification unit that uses the classifier to identify the position of the reducing lane in response to the input of the sign portion. The vehicle control device according to claim 2, wherein the determination unit determines the position of the reducing lane based on at least one of the determination result of the first determination unit and the determination result of the second determination unit and the determination result of the third determination unit.

4. The vehicle control device according to claim 1, wherein the lane reduction identification unit identifies the leftmost lane as the lane reduction when the first shape is located at the left end of the sign portion, and identifies the rightmost lane as the lane reduction when the first shape is located at the right end of the sign portion.

5. The vehicle control device according to claim 4, wherein the lane reduction identification unit identifies the position of the lane reduction based on the position of the first shape of a second shape among a plurality of second shapes, the second shape having a vertical length less than or equal to a predetermined value relative to the vertical length of the first shape.

6. A vehicle control device according to any one of claims 1 to 5, further comprising a lane determination unit that determines whether the lane in which the vehicle is traveling is the aforementioned lane reduction lane, based on the lane reduction information relating to the lane reduction output by the lane reduction identification unit and the position of the vehicle.

7. The vehicle control device according to claim 6, further comprising a notification unit that controls a notification device in order to notify the occupants when the lane determination unit determines that the lane in which the vehicle is traveling is the narrowing lane.

8. The vehicle control device according to claim 6, further comprising a driving support unit that controls the steering device to cause the vehicle to change lanes to a different remaining lane when the lane determination unit determines that the lane in which the vehicle is traveling is the narrowing lane.

9. The vehicle control device according to claim 6, wherein the lane reduction identification unit identifies the lane reduction at predetermined time intervals and corrects the current identification result based on the previous identification result.

10. A vehicle control method performed by a computer, From the forward image captured by a camera that captures the area around the vehicle, the system recognizes a sign portion corresponding to a lane reduction sign, which includes one or two folded line sections including a bend, and multiple straight sections that are solid or dashed lines. A vehicle control method that obtains one or two first shapes corresponding to one or two of the folded line portions and a plurality of second shapes corresponding to a plurality of the straight sections from the sign portion, and determines the position of the reducing lane based on the one or two first shapes and the plurality of second shapes.

11. A control program that causes a computer to execute a vehicle control method, From the forward image captured by a camera that captures the area around the vehicle, the system recognizes a sign portion corresponding to a lane reduction sign, which includes one or two broken lines including a bend, and multiple straight lines that are solid or dashed lines. A control program that obtains one or two first shapes corresponding to one or two of the folded line sections and a plurality of second shapes corresponding to a plurality of the straight sections from the sign portion, and determines the position of the reducing lane based on the one or two first shapes and the plurality of second shapes.

Images